Sep 11 21:50:33.664305 ip-10-0-128-201 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 21:50:33.664315 ip-10-0-128-201 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 21:50:33.664322 ip-10-0-128-201 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 21:50:33.664629 ip-10-0-128-201 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 21:50:42.530432 ip-10-0-128-201 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 21:50:42.530442 ip-10-0-128-201 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 70fd7fd10a0648bcb57211f02dee3082 --
Sep 11 21:52:13.155686 ip-10-0-128-201 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 21:52:13.532969 ip-10-0-128-201 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:52:13.532969 ip-10-0-128-201 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 21:52:13.532969 ip-10-0-128-201 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:52:13.532969 ip-10-0-128-201 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 21:52:13.532969 ip-10-0-128-201 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:52:13.533628 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.532999    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 21:52:13.536709 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536692    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:13.536709 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536708    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536712    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536716    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536719    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536722    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536725    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536727    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536730    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536733    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536736    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536738    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536742    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536745    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536747    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536750    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536753    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536755    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536758    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536760    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:13.536773 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536765    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536769    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536772    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536775    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536779    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536782    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536786    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536789    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536792    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536795    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536798    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536801    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536804    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536806    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536809    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536812    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536814    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536817    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536819    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536821    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:13.537262 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536824    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536826    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536829    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536831    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536834    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536836    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536839    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536842    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536845    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536848    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536850    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536853    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536855    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536858    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536861    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536864    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536866    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536869    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536872    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:13.537758 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536876    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536878    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536881    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536883    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536886    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536889    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536892    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536895    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536899    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536902    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536905    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536908    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536911    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536914    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536917    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536919    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536922    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536925    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536928    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:13.538232 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536930    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536933    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536935    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536938    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536941    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536944    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536946    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536949    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536953    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536955    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536960    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536964    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536967    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536970    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536973    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536975    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536978    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536980    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536983    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536987    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:13.538699 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.536989    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537713    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537720    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537724    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537727    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537731    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537733    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537736    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537740    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537744    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537747    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537750    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537753    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537756    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537759    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537763    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537766    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537769    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537771    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537775    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:13.539188 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537779    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537781    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537784    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537787    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537790    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537792    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537795    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537797    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537800    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537803    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537805    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537808    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537811    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537814    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537816    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537819    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537821    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537824    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537827    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537829    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:13.539689 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537831    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537834    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537837    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537839    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537842    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537844    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537847    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537849    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537852    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537854    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537857    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537860    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537863    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537866    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537868    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537871    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537874    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537876    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537879    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537881    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:13.540183 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537885    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537887    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537890    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537892    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537895    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537898    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537900    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537903    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537905    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537908    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537910    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537913    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537915    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537918    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537920    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537923    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537925    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537928    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537930    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:13.540693 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537933    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537935    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537938    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537940    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537943    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537947    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537951    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537954    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537957    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537959    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537963    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537968    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537972    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537974    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537977    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537979    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537982    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537984    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537987    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:13.541175 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537990    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.537993    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538349    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538354    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538357    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538359    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538362    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538365    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538367    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538370    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538373    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538376    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538379    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538381    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538384    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538386    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538389    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538392    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538394    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538397    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:13.541656 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538400    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538403    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538405    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538408    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538411    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538414    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538416    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538419    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538421    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538424    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538426    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538429    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538432    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538435    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538437    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538440    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538442    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538445    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538447    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:13.542144 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538449    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538452    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538455    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538457    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538460    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538462    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538465    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538468    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538470    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538473    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538475    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538478    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538480    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538482    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538485    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538488    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538490    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538493    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538495    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538498    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:13.542666 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538501    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538505    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538508    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538511    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538514    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538517    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538523    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538526    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538529    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538531    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538534    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538536    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538539    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538541    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538544    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538547    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538549    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538551    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538556    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538559    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:13.543168 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538562    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538564    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538567    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538570    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538574    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538576    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538579    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538581    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538584    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538587    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538589    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538593    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538597    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538601    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538604    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538607    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538610    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538612    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538620    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:13.543719 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538622    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538625    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.538627    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538702    2574 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538737    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538743    2574 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538747    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538751    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538754    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538759    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538763    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538767    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538770    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538774    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538777    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538780    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538783    2574 flags.go:64] FLAG: --cgroup-root=""
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538786    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538789    2574 flags.go:64] FLAG: --client-ca-file=""
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538792    2574 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538795    2574 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538800    2574 flags.go:64] FLAG: --cluster-domain=""
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538803    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538806    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 21:52:13.544178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538809    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538812    2574 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538816    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538819    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538822    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538825    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538828    2574 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538832    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538835    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538840    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538843    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538847    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538850    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538853    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538856    2574 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538859    2574 flags.go:64] FLAG: --enable-server="true"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538862    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538866    2574 flags.go:64] FLAG: --event-burst="100"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538869    2574 flags.go:64] FLAG: --event-qps="50"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538873    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538876    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538879    2574 flags.go:64] FLAG: --eviction-hard=""
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538882    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538885    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538888    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 21:52:13.544770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538892    2574 flags.go:64] FLAG: --eviction-soft=""
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538895    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538898    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538901    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538904    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538907    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538909    2574 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538912    2574 flags.go:64] FLAG: --feature-gates=""
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538921    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538924    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538927    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538930    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538933    2574 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538936    2574 flags.go:64] FLAG: --help="false"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538940    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-128-201.ec2.internal"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538943    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538946    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538950    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538953    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538956    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538959    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538962    2574 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538965    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 21:52:13.545496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538968    2574 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538971    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538974    2574 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538978    2574 flags.go:64] FLAG: --kube-reserved=""
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538981    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538984    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538987    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538990    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538992    2574 flags.go:64] FLAG: --lock-file=""
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538995    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.538998    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539001    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539007    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539010    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539013    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539016    2574 flags.go:64] FLAG: --logging-format="text"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539018    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539022    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539025    2574 flags.go:64] FLAG: --manifest-url=""
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539027    2574 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539031    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539035    2574 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539039    2574 flags.go:64] FLAG: --max-pods="110"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539042    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539045    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 21:52:13.546098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539048    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539051    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539055    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539058    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539061    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539068    2574 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539071    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539074    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539077    2574 flags.go:64] FLAG: --pod-cidr=""
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539080    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539086    2574 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539090    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539093    2574 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539096    2574 flags.go:64] FLAG: --port="10250"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539099    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539103    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f59a626268bf7129"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539106    2574 flags.go:64] FLAG: --qos-reserved=""
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539109    2574 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539112    2574 flags.go:64] FLAG: --register-node="true"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539115    2574 flags.go:64] FLAG: --register-with-taints=""
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539119    2574 flags.go:64] FLAG: --registry-burst="10"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539122    2574 flags.go:64] FLAG: --registry-qps="5"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539124    2574 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539127    2574 flags.go:64] FLAG: --reserved-memory=""
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539131    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 21:52:13.546900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539134    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539137    2574 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539140    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539143    2574 flags.go:64] FLAG: --runonce="false"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539146    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539150    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539153    2574 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539156    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539159    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539162    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539166    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539169    2574 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539171    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539174    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539177    2574 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539180    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539183    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539186    2574 flags.go:64] FLAG: --system-cgroups=""
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539190    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539196    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539199    2574 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539202    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539206    2574 flags.go:64] FLAG: --tls-min-version=""
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539209    2574 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539212    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 21:52:13.547706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539215    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539218    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539221    2574 flags.go:64] FLAG: --v="2"
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539225    2574 flags.go:64] FLAG: --version="false"
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539228    2574 flags.go:64] FLAG: --vmodule=""
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539232    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.539235    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539380    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539385    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539388    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539391    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539394    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539400    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539403    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539406    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539408    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539411    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539414    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539418    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539420    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539423    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:13.548335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539426    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539428    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539431    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539434    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539438    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539441    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539444    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539446    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539449    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539451    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539454    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539457    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539459    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539462    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539466    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539469    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539472    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539475    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539478    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539480    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:13.548849 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539483    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539485    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539488    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539492    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539495    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539497    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539500    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539502    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539505    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539508    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539511    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539513    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539516    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539519    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539521    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539524    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539527    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539530    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539533    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539535    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:13.549358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539538    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539541    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539543    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539546    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539548    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539551    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539553    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539556    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539559    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539562    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539564    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539567    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539569    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539572    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539574    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539578    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539583    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539586    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539590    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:13.549852 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539593    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539597    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539601    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539606    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539611    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539615    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539618    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539620    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539623    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539627    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539630    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539632    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539635    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539637    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539640    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539642    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539645    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539647    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539650    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539653    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:13.550484 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539656    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539658    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539661    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539663    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539666    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.539668    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.540285    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.548871    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.548884    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.548910    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.548916    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.548992    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.548996    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549000    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549003    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:13.550984 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549005    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549008    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549011    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549014    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549016    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549019    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549022    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549025    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549027    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549030    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549033    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549035    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549038    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549041    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549043    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549046    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549049    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549051    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549054    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549057    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:13.551386 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549059    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549061    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549064    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549067    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549070    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549074    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549077    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549080    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549082    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549085    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549088    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549091    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549093    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549096    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549098    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549102    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549106    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549108    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549111    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:13.551865 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549113    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549116    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549119    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549122    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549124    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549127    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549130    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549133    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549135    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549138    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549141    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549143    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549146    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549148    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549151    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549153    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549156    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549159    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549162    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549166    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:13.552364 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549170    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549173    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549176    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549179    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549182    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549185    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549188    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549190    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549193    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549196    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549198    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549201    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549204    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549206    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549209    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549211    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549215    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549218    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549221    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549223    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:13.552942 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549226    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549229    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549232    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549235    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549237    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549240    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549243    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549246    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549249    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549252    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549256    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549260    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549263    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549266    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549268    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:13.553491 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549271    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.549276    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549417    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549423    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549426    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549430    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549435    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549438    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549441    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549443    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549446    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549449    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549451    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549454    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549457    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:13.553983 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549459    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549462    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549465    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549468    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549471    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549474    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549476    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549479    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549481    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549484    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549486    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549489    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549491    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549495    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549498    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549500    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549503    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549506    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549508    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549511    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:13.554640 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549514    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549516    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549519    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549521    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549524    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549528    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549531    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549534    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549537    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549540    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549542    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549545    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549549    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549551    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549554    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549557    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549560    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549563    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549565    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549568    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:13.555236 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549570    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549573    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549576    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549578    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549581    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549583    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549586    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549588    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549591    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549593    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549596    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549598    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549601    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549604    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549606    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549609    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549611    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549614    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549617    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549619    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:13.555842 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549622    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549625    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549627    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549630    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549632    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549635    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549638    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549640    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549643    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549646    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549648    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549651    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549653    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549656    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549658    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549662    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549665    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549668    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549671    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:13.556341 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549674    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:13.556812 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549676    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:13.556812 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549679    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:13.556812 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549682    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:13.556812 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549685    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:13.556812 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549687    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:13.556812 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:13.549690    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:13.556812 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.549694    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:52:13.556812 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.549805    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 21:52:13.556812 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.551861    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 21:52:13.556812 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.552701    2574 server.go:1019] "Starting client certificate rotation"
Sep 11 21:52:13.556812 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.552812    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:52:13.556812 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.552868    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:52:13.556812 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.554969    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:52:13.557819 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.557803    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:52:13.570170 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.570155    2574 log.go:25] "Validated CRI v1 runtime API"
Sep 11 21:52:13.573792 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.573777    2574 log.go:25] "Validated CRI v1 image API"
Sep 11 21:52:13.574630 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.574613    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 21:52:13.577690 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.577672    2574 fs.go:135] Filesystem UUIDs: map[45bfb655-dd01-448d-bfa5-01c3944396b4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e23e8c58-afe2-4af4-a14d-aa01086810c4:/dev/nvme0n1p4]
Sep 11 21:52:13.577743 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.577691    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 21:52:13.583398 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.583280    2574 manager.go:216] Machine: {Timestamp:2026-09-11 21:52:13.582118497 +0000 UTC m=+0.317229837 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098266 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24459bc7d5f6dc7754ec35052c1aae SystemUUID:ec24459b-c7d5-f6dc-7754-ec35052c1aae BootID:70fd7fd1-0a06-48bc-b572-11f02dee3082 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:32:b8:6f:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:32:b8:6f:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:7e:e7:1e:61:a8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 21:52:13.583398 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.583393    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 21:52:13.583537 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.583487    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 21:52:13.583591 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.583564    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:52:13.587536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.587514    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 21:52:13.587665 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.587538    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-201.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 21:52:13.588301 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.588292    2574 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 21:52:13.588347 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.588304    2574 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 21:52:13.588347 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.588325    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:52:13.589125 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.589115    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:52:13.590028 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.590019    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 21:52:13.593938 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.593926    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:52:13.594042 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.594033    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 21:52:13.594081 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.594061    2574 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 21:52:13.594081 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.594070    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 21:52:13.594137 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.594084    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 21:52:13.594137 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.594094    2574 kubelet.go:398] "Adding apiserver pod source"
Sep 11 21:52:13.594137 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.594103    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 21:52:13.594536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.594521    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:52:13.594536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.594545    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:52:13.596754 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.596739    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 21:52:13.599197 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.599173    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 21:52:13.599273 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.599218    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 21:52:13.600702 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.600688    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 21:52:13.600789 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.600709    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 21:52:13.600789 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.600718    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 21:52:13.600789 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.600742    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 21:52:13.600789 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.600757    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 21:52:13.600789 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.600766    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 21:52:13.600789 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.600787    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 21:52:13.601028 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.600796    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 21:52:13.601028 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.600849    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 21:52:13.601028 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.600865    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 21:52:13.601028 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.600890    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 21:52:13.601028 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.600900    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 21:52:13.601528 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.601517    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 21:52:13.601591 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.601530    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 21:52:13.604781 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.604762    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-201.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 21:52:13.605442 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.605403    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 21:52:13.605490 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.605423    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 21:52:13.605533 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.605493    2574 server.go:1268] "Started kubelet"
Sep 11 21:52:13.605649 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.605599    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 21:52:13.605742 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.605668    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 21:52:13.605742 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.605638    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 21:52:13.606365 ip-10-0-128-201 systemd[1]: Started Kubernetes Kubelet.
Sep 11 21:52:13.606783 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.606764    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 21:52:13.611383 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.611358    2574 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 21:52:13.611602 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.611579    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 21:52:13.614605 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.613774    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-201.ec2.internal.18d462c121a2f347  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-201.ec2.internal,UID:ip-10-0-128-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-201.ec2.internal,},FirstTimestamp:2026-09-11 21:52:13.605466951 +0000 UTC m=+0.340578288,LastTimestamp:2026-09-11 21:52:13.605466951 +0000 UTC m=+0.340578288,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-201.ec2.internal,}"
Sep 11 21:52:13.615170 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.615150    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 21:52:13.615671 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.615655    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 21:52:13.615993 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.615880    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found"
Sep 11 21:52:13.616649 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.616338    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 21:52:13.616649 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.616337    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 21:52:13.616649 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.616375    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 21:52:13.616649 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.616461    2574 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 21:52:13.616649 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.616472    2574 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 21:52:13.616649 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.616463    2574 factory.go:55] Registering systemd factory
Sep 11 21:52:13.616649 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.616493    2574 factory.go:223] Registration of the systemd container factory successfully
Sep 11 21:52:13.616891 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.616752    2574 factory.go:153] Registering CRI-O factory
Sep 11 21:52:13.616891 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.616766    2574 factory.go:223] Registration of the crio container factory successfully
Sep 11 21:52:13.616891 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.616812    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 21:52:13.616891 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.616829    2574 factory.go:103] Registering Raw factory
Sep 11 21:52:13.616891 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.616844    2574 manager.go:1196] Started watching for new ooms in manager
Sep 11 21:52:13.617301 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.617289    2574 manager.go:349] Starting recovery of all containers
Sep 11 21:52:13.618399 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.618372    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 21:52:13.618499 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.618413    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 21:52:13.628963 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.628903    2574 manager.go:354] Recovery completed
Sep 11 21:52:13.631277 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.631253    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lbtgl"
Sep 11 21:52:13.632850 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.632837    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:13.635184 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.635166    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:13.635231 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.635202    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:13.635231 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.635217    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:13.637276 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.637014    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:13.637388 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.637284    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 21:52:13.637388 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.637299    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 21:52:13.637388 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.637332    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:52:13.637671 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.637582    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-201.ec2.internal.18d462c123686a2e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-201.ec2.internal,UID:ip-10-0-128-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-201.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-201.ec2.internal,},FirstTimestamp:2026-09-11 21:52:13.635185198 +0000 UTC m=+0.370296540,LastTimestamp:2026-09-11 21:52:13.635185198 +0000 UTC m=+0.370296540,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-201.ec2.internal,}"
Sep 11 21:52:13.638039 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.638020    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lbtgl"
Sep 11 21:52:13.638257 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.638239    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:13.638328 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.638269    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:13.638328 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.638280    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:13.639483 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.639470    2574 policy_none.go:49] "None policy: Start"
Sep 11 21:52:13.639532 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.639485    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 21:52:13.639532 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.639497    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 21:52:13.641383 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.641369    2574 policy_none.go:47] "Start"
Sep 11 21:52:13.698982 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.675610    2574 manager.go:337] "Starting Device Plugin manager"
Sep 11 21:52:13.698982 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.675641    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 21:52:13.698982 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.675651    2574 server.go:87] "Starting device plugin registration server"
Sep 11 21:52:13.698982 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.675857    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 21:52:13.698982 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.675868    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 21:52:13.698982 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.675940    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 21:52:13.698982 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.676008    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 21:52:13.698982 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.676015    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 21:52:13.698982 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.676551    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 21:52:13.698982 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.676617    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-201.ec2.internal\" not found"
Sep 11 21:52:13.712226 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.712192    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 21:52:13.713476 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.713456    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 21:52:13.713554 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.713501    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 21:52:13.713554 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.713525    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 21:52:13.713841 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.713614    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 21:52:13.715754 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.715734    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:13.776565 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.776524    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:13.777524 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.777508    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:13.777585 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.777543    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:13.777585 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.777558    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:13.777651 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.777588    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-201.ec2.internal"
Sep 11 21:52:13.788380 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.788332    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-201.ec2.internal"
Sep 11 21:52:13.788380 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.788354    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-201.ec2.internal\": node \"ip-10-0-128-201.ec2.internal\" not found"
Sep 11 21:52:13.810562 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.810543    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found"
Sep 11 21:52:13.814701 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.814681    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal"]
Sep 11 21:52:13.814780 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.814770    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:13.816287 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.816271    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:13.816411 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.816298    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:13.816411 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.816322    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:13.817402 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.817385    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba1918520a962a77d1f14b1726c5bd1b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal\" (UID: \"ba1918520a962a77d1f14b1726c5bd1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:13.817442 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.817414    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1918520a962a77d1f14b1726c5bd1b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal\" (UID: \"ba1918520a962a77d1f14b1726c5bd1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:13.818465 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.818444    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:13.818603 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.818588    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:13.818666 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.818623    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:13.819153 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.819135    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:13.819232 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.819167    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:13.819232 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.819178    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:13.819232 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.819137    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:13.819398 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.819246    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:13.819398 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.819263    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:13.821990 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.821974    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:13.822072 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.822003    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:13.822660 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.822644    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:13.822722 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.822675    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:13.822722 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.822689    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:13.845017 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.845002    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-201.ec2.internal\" not found" node="ip-10-0-128-201.ec2.internal"
Sep 11 21:52:13.849569 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.849552    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-201.ec2.internal\" not found" node="ip-10-0-128-201.ec2.internal"
Sep 11 21:52:13.910840 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:13.910822    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found"
Sep 11 21:52:13.918183 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.918167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba1918520a962a77d1f14b1726c5bd1b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal\" (UID: \"ba1918520a962a77d1f14b1726c5bd1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:13.918233 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.918188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1918520a962a77d1f14b1726c5bd1b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal\" (UID: \"ba1918520a962a77d1f14b1726c5bd1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:13.918233 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.918211    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91e28972004d554b545a91ff30beb546-config\") pod \"kube-apiserver-proxy-ip-10-0-128-201.ec2.internal\" (UID: \"91e28972004d554b545a91ff30beb546\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:13.918293 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.918271    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1918520a962a77d1f14b1726c5bd1b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal\" (UID: \"ba1918520a962a77d1f14b1726c5bd1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:13.918293 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:13.918273    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba1918520a962a77d1f14b1726c5bd1b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal\" (UID: \"ba1918520a962a77d1f14b1726c5bd1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:14.011354 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.011323    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found"
Sep 11 21:52:14.018628 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.018610    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91e28972004d554b545a91ff30beb546-config\") pod \"kube-apiserver-proxy-ip-10-0-128-201.ec2.internal\" (UID: \"91e28972004d554b545a91ff30beb546\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:14.018628 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.018619    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91e28972004d554b545a91ff30beb546-config\") pod \"kube-apiserver-proxy-ip-10-0-128-201.ec2.internal\" (UID: \"91e28972004d554b545a91ff30beb546\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:14.112094 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.112037    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found"
Sep 11 21:52:14.148528 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.148509    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:14.152168 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.152147    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:14.212823 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.212801    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found"
Sep 11 21:52:14.313279 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.313255    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found"
Sep 11 21:52:14.414292 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.414264    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found"
Sep 11 21:52:14.435720 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.435695    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:14.514920 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.514889    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found"
Sep 11 21:52:14.518299 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.518278    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:14.555130 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.555103    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 21:52:14.555795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.555225    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:52:14.555795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.555253    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:52:14.555795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.555266    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:52:14.594305 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.594273    2574 apiserver.go:52] "Watching apiserver"
Sep 11 21:52:14.612915 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.612891    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 21:52:14.613260 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.613239    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-frptb","openshift-multus/multus-qcn7v","openshift-network-operator/iptables-alerter-g2swb","openshift-ovn-kubernetes/ovnkube-node-wzl98","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb","openshift-dns/node-resolver-js4ll","openshift-multus/network-metrics-daemon-9swth","openshift-network-diagnostics/network-check-target-5h829","kube-system/global-pull-secret-syncer-628px","kube-system/konnectivity-agent-52qz9","openshift-cluster-node-tuning-operator/tuned-wzp59","openshift-image-registry/node-ca-vhjjf"]
Sep 11 21:52:14.615287 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.615259    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 21:52:14.616599 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.616250    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:14.616599 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.616303    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.619334 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.619315    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 21:52:14.619627 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.619611    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 21:52:14.619790 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.619774    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 21:52:14.619898 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.619882    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xbbr2\""
Sep 11 21:52:14.620035 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.620021    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 21:52:14.620157 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.620141    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.620838 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.620816    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 21:52:14.621681 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.621660    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6071a312-2b91-4da6-aaf0-993a791e109f-multus-daemon-config\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.621788 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.621685    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-etc-kubernetes\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.621788 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.621704    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpbdf\" (UniqueName: \"kubernetes.io/projected/6071a312-2b91-4da6-aaf0-993a791e109f-kube-api-access-dpbdf\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.621788 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.621727    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c735de58-8252-4bfe-b52d-5b1682fe92b8-cni-binary-copy\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.621788 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.621747    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c735de58-8252-4bfe-b52d-5b1682fe92b8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.621947 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.621811    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-system-cni-dir\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.621947 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.621837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-multus-socket-dir-parent\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.621947 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.621856    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-hostroot\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.621947 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.621881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-run-multus-certs\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.621947 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.621903    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c735de58-8252-4bfe-b52d-5b1682fe92b8-system-cni-dir\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.621947 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.621944    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c735de58-8252-4bfe-b52d-5b1682fe92b8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.622157 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.621969    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c735de58-8252-4bfe-b52d-5b1682fe92b8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.622157 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.621995    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdffs\" (UniqueName: \"kubernetes.io/projected/c735de58-8252-4bfe-b52d-5b1682fe92b8-kube-api-access-tdffs\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.622157 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.622099    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-multus-cni-dir\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.622157 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.622126    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-var-lib-cni-multus\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.622368 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.622148    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-multus-conf-dir\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.622368 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.622225    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6071a312-2b91-4da6-aaf0-993a791e109f-cni-binary-copy\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.622368 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.622286    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-run-k8s-cni-cncf-io\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.622368 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.622343    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-var-lib-cni-bin\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.622531 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.622376    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-var-lib-kubelet\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.622531 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.622400    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c735de58-8252-4bfe-b52d-5b1682fe92b8-cnibin\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.622531 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.622428    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-cnibin\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.622531 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.622444    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-run-netns\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.622707 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.622622    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 21:52:14.622884 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.622866    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g2swb"
Sep 11 21:52:14.623733 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.623715    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4mr4m\""
Sep 11 21:52:14.625380 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.625363    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:52:14.625618 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.625602    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6lrps\""
Sep 11 21:52:14.625711 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.625653    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 21:52:14.625711 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.625676    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 21:52:14.625872 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.625854    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.626334 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.626312    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:52:14.628519 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.628499    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tscts\""
Sep 11 21:52:14.628776 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.628755    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 21:52:14.628853 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.628635    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 21:52:14.628910 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.628865    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 21:52:14.629002 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.628979    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 21:52:14.629110 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.629045    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.630068 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.630048    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 21:52:14.631008 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.630983    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 21:52:14.631678 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.631658    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-js4ll"
Sep 11 21:52:14.632113 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.632096    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 21:52:14.632214 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.632120    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r5nrg\""
Sep 11 21:52:14.632214 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.632136    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 21:52:14.632214 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.632150    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 21:52:14.632522 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.632508    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:52:14.632592 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.632581    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal"
Sep 11 21:52:14.634343 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.634326    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 21:52:14.634487 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.634442    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 21:52:14.634590 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.634573    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-24tmj\""
Sep 11 21:52:14.634966 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.634950    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:14.635063 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.635033    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:14.637965 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.637942    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:14.638098 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.638010    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:14.641046 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.640999    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:14.641136 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.641064    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:14.643243 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.643207    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 21:47:14 +0000 UTC" deadline="2028-04-18 21:15:05.039704789 +0000 UTC"
Sep 11 21:52:14.643305 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.643243    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14039h22m50.39646574s"
Sep 11 21:52:14.643305 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.643248    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-52qz9"
Sep 11 21:52:14.643913 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.643893    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:52:14.645803 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.645784    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.646373 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.646353    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 21:52:14.646482 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.646408    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2gpf6\""
Sep 11 21:52:14.646482 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.646460    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 21:52:14.648348 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.648171    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal"]
Sep 11 21:52:14.648348 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.648211    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal"]
Sep 11 21:52:14.648348 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.648264    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vhjjf"
Sep 11 21:52:14.648573 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.648511    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kx7x5\""
Sep 11 21:52:14.648573 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.648527    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:52:14.649301 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.648874    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 21:52:14.651082 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.651065    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 21:52:14.651553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.651537    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5x992\""
Sep 11 21:52:14.651963 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.651945    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 21:52:14.652031 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.651943    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 21:52:14.655556 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.655539    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dj56b"
Sep 11 21:52:14.664022 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.663799    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dj56b"
Sep 11 21:52:14.717472 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.717455    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 21:52:14.723322 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723300    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-var-lib-kubelet\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.723405 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723341    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:14.723405 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723362    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae03deae-80f3-4af0-b22c-edd9123df3ff-tmp\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.723405 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723377    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-device-dir\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.723405 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723400    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-kubelet\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.723541 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723438    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.723541 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723503    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-var-lib-kubelet\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.723541 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723526    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-ovnkube-script-lib\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.723633 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c735de58-8252-4bfe-b52d-5b1682fe92b8-cnibin\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.723633 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-run-netns\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.723633 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723622    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c735de58-8252-4bfe-b52d-5b1682fe92b8-cnibin\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.723728 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723633    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpbdf\" (UniqueName: \"kubernetes.io/projected/6071a312-2b91-4da6-aaf0-993a791e109f-kube-api-access-dpbdf\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.723728 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723646    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-run-netns\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.723728 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723670    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39564d9a-8271-4a01-bfe2-8e7c30c227b2-agent-certs\") pod \"konnectivity-agent-52qz9\" (UID: \"39564d9a-8271-4a01-bfe2-8e7c30c227b2\") " pod="kube-system/konnectivity-agent-52qz9"
Sep 11 21:52:14.723728 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723685    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-var-lib-openvswitch\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.723728 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723704    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jftm7\" (UniqueName: \"kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7\") pod \"network-check-target-5h829\" (UID: \"ec958662-21bf-4b7a-89e8-267265fec58f\") " pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:14.723885 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723739    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-host\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.723885 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723765    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fedd4091-aaa3-44e1-9643-2a3e5c30be81-host-slash\") pod \"iptables-alerter-g2swb\" (UID: \"fedd4091-aaa3-44e1-9643-2a3e5c30be81\") " pod="openshift-network-operator/iptables-alerter-g2swb"
Sep 11 21:52:14.723885 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723795    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.723885 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-ovn-node-metrics-cert\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.723885 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723838    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c735de58-8252-4bfe-b52d-5b1682fe92b8-system-cni-dir\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.723885 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723867    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c735de58-8252-4bfe-b52d-5b1682fe92b8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.724187 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723887    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c735de58-8252-4bfe-b52d-5b1682fe92b8-system-cni-dir\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.724187 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723918    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-sysconfig\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.724187 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fbd851a8-185e-4b99-bcb6-76c0d1c63f1c-serviceca\") pod \"node-ca-vhjjf\" (UID: \"fbd851a8-185e-4b99-bcb6-76c0d1c63f1c\") " pod="openshift-image-registry/node-ca-vhjjf"
Sep 11 21:52:14.724187 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723968    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl55q\" (UniqueName: \"kubernetes.io/projected/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-kube-api-access-cl55q\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.724187 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.723996    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-tuned\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.724187 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.724187 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724085    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-registration-dir\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.724187 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724113    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-895lc\" (UniqueName: \"kubernetes.io/projected/fbd851a8-185e-4b99-bcb6-76c0d1c63f1c-kube-api-access-895lc\") pod \"node-ca-vhjjf\" (UID: \"fbd851a8-185e-4b99-bcb6-76c0d1c63f1c\") " pod="openshift-image-registry/node-ca-vhjjf"
Sep 11 21:52:14.724187 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724142    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-etc-openvswitch\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.724187 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-dbus\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:14.724187 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-modprobe-d\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724226    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-run-ovn\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724262    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-multus-socket-dir-parent\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724279    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-sysctl-d\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724303    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-multus-socket-dir-parent\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724322    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c735de58-8252-4bfe-b52d-5b1682fe92b8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-run\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724427    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdffs\" (UniqueName: \"kubernetes.io/projected/c735de58-8252-4bfe-b52d-5b1682fe92b8-kube-api-access-tdffs\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724454    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fbd851a8-185e-4b99-bcb6-76c0d1c63f1c-host\") pod \"node-ca-vhjjf\" (UID: \"fbd851a8-185e-4b99-bcb6-76c0d1c63f1c\") " pod="openshift-image-registry/node-ca-vhjjf"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-systemd-units\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724513    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-run-netns\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724547    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-run-openvswitch\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724582    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-kubernetes\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724603    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-systemd\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.724626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724619    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-ovnkube-config\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724641    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-cnibin\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724661    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-etc-kubernetes\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724678    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-sysctl-conf\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724693    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-cnibin\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724694    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-system-cni-dir\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724722    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-etc-kubernetes\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-run-multus-certs\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724806    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-system-cni-dir\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724833    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv922\" (UniqueName: \"kubernetes.io/projected/f1c80315-39cc-4397-be35-3d7de7f7e9a1-kube-api-access-bv922\") pod \"node-resolver-js4ll\" (UID: \"f1c80315-39cc-4397-be35-3d7de7f7e9a1\") " pod="openshift-dns/node-resolver-js4ll"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724847    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-run-systemd\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724852    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-run-multus-certs\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724895    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-env-overrides\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724929    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-multus-cni-dir\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-var-lib-cni-multus\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.724982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-multus-conf-dir\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725037    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-var-lib-cni-multus\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725038    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fedd4091-aaa3-44e1-9643-2a3e5c30be81-iptables-alerter-script\") pod \"iptables-alerter-g2swb\" (UID: \"fedd4091-aaa3-44e1-9643-2a3e5c30be81\") " pod="openshift-network-operator/iptables-alerter-g2swb"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725048    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-multus-cni-dir\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725055    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-multus-conf-dir\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725067    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c80315-39cc-4397-be35-3d7de7f7e9a1-tmp-dir\") pod \"node-resolver-js4ll\" (UID: \"f1c80315-39cc-4397-be35-3d7de7f7e9a1\") " pod="openshift-dns/node-resolver-js4ll"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725087    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72dxs\" (UniqueName: \"kubernetes.io/projected/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-kube-api-access-72dxs\") pod \"network-metrics-daemon-9swth\" (UID: \"f07461d1-42d3-43f6-bf6a-89a8663f4ca8\") " pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725101    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-cni-bin\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725141    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6071a312-2b91-4da6-aaf0-993a791e109f-cni-binary-copy\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725175    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-var-lib-cni-bin\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725204    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-var-lib-cni-bin\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725213    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-kubelet-config\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725242    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39564d9a-8271-4a01-bfe2-8e7c30c227b2-konnectivity-ca\") pod \"konnectivity-agent-52qz9\" (UID: \"39564d9a-8271-4a01-bfe2-8e7c30c227b2\") " pod="kube-system/konnectivity-agent-52qz9"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-slash\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725296    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6071a312-2b91-4da6-aaf0-993a791e109f-multus-daemon-config\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725320    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-sys\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725362    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sgx9\" (UniqueName: \"kubernetes.io/projected/fedd4091-aaa3-44e1-9643-2a3e5c30be81-kube-api-access-2sgx9\") pod \"iptables-alerter-g2swb\" (UID: \"fedd4091-aaa3-44e1-9643-2a3e5c30be81\") " pod="openshift-network-operator/iptables-alerter-g2swb"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725388    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-sys-fs\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725420    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhg9w\" (UniqueName: \"kubernetes.io/projected/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-kube-api-access-bhg9w\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.725972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725446    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs\") pod \"network-metrics-daemon-9swth\" (UID: \"f07461d1-42d3-43f6-bf6a-89a8663f4ca8\") " pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725496    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c735de58-8252-4bfe-b52d-5b1682fe92b8-cni-binary-copy\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725528    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c735de58-8252-4bfe-b52d-5b1682fe92b8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725546    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-hostroot\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725580    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6071a312-2b91-4da6-aaf0-993a791e109f-cni-binary-copy\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725621    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-hostroot\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-lib-modules\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725669    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-var-lib-kubelet\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725695    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-socket-dir\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725714    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c735de58-8252-4bfe-b52d-5b1682fe92b8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725720    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-node-log\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725717    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6071a312-2b91-4da6-aaf0-993a791e109f-multus-daemon-config\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725745    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-cni-netd\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c735de58-8252-4bfe-b52d-5b1682fe92b8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725790    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49kb\" (UniqueName: \"kubernetes.io/projected/ae03deae-80f3-4af0-b22c-edd9123df3ff-kube-api-access-j49kb\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725806    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1c80315-39cc-4397-be35-3d7de7f7e9a1-hosts-file\") pod \"node-resolver-js4ll\" (UID: \"f1c80315-39cc-4397-be35-3d7de7f7e9a1\") " pod="openshift-dns/node-resolver-js4ll"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725826    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-log-socket\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.726553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725851    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-run-k8s-cni-cncf-io\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.727059 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.725919    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6071a312-2b91-4da6-aaf0-993a791e109f-host-run-k8s-cni-cncf-io\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.727059 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.726025    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c735de58-8252-4bfe-b52d-5b1682fe92b8-cni-binary-copy\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.727059 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.726157    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c735de58-8252-4bfe-b52d-5b1682fe92b8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.730893 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.730862    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 21:52:14.734180 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.734153    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdffs\" (UniqueName: \"kubernetes.io/projected/c735de58-8252-4bfe-b52d-5b1682fe92b8-kube-api-access-tdffs\") pod \"multus-additional-cni-plugins-frptb\" (UID: \"c735de58-8252-4bfe-b52d-5b1682fe92b8\") " pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.734279 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.734225    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpbdf\" (UniqueName: \"kubernetes.io/projected/6071a312-2b91-4da6-aaf0-993a791e109f-kube-api-access-dpbdf\") pod \"multus-qcn7v\" (UID: \"6071a312-2b91-4da6-aaf0-993a791e109f\") " pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.804451 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:14.804424    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e28972004d554b545a91ff30beb546.slice/crio-201a2b716ea7ca981308b606a7e88e44c7679cb0a837067cc67d38f8cdf2f443 WatchSource:0}: Error finding container 201a2b716ea7ca981308b606a7e88e44c7679cb0a837067cc67d38f8cdf2f443: Status 404 returned error can't find the container with id 201a2b716ea7ca981308b606a7e88e44c7679cb0a837067cc67d38f8cdf2f443
Sep 11 21:52:14.804722 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:14.804701    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1918520a962a77d1f14b1726c5bd1b.slice/crio-3679f7168f017511dd8a41463987cd8272e7ae3ed9bfdbcddd2a566dfe950990 WatchSource:0}: Error finding container 3679f7168f017511dd8a41463987cd8272e7ae3ed9bfdbcddd2a566dfe950990: Status 404 returned error can't find the container with id 3679f7168f017511dd8a41463987cd8272e7ae3ed9bfdbcddd2a566dfe950990
Sep 11 21:52:14.809952 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.809938    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:52:14.826440 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826419    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:14.826541 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826496    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae03deae-80f3-4af0-b22c-edd9123df3ff-tmp\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.826541 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-device-dir\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.826664 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.826570    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:14.826664 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-kubelet\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.826664 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.826626    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret podName:b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa nodeName:}" failed. No retries permitted until 2026-09-11 21:52:15.326610297 +0000 UTC m=+2.061721622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret") pod "global-pull-secret-syncer-628px" (UID: "b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:14.826664 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.826664 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826658    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-ovnkube-script-lib\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.826954 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826687    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39564d9a-8271-4a01-bfe2-8e7c30c227b2-agent-certs\") pod \"konnectivity-agent-52qz9\" (UID: \"39564d9a-8271-4a01-bfe2-8e7c30c227b2\") " pod="kube-system/konnectivity-agent-52qz9"
Sep 11 21:52:14.826954 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826713    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-var-lib-openvswitch\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.826954 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826716    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.826954 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftm7\" (UniqueName: \"kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7\") pod \"network-check-target-5h829\" (UID: \"ec958662-21bf-4b7a-89e8-267265fec58f\") " pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:14.826954 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826736    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-kubelet\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.826954 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-host\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.826954 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826807    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fedd4091-aaa3-44e1-9643-2a3e5c30be81-host-slash\") pod \"iptables-alerter-g2swb\" (UID: \"fedd4091-aaa3-44e1-9643-2a3e5c30be81\") " pod="openshift-network-operator/iptables-alerter-g2swb"
Sep 11 21:52:14.826954 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826834    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.826954 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826912    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-ovn-node-metrics-cert\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.826954 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826953    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-sysconfig\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.827569 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.826985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fbd851a8-185e-4b99-bcb6-76c0d1c63f1c-serviceca\") pod \"node-ca-vhjjf\" (UID: \"fbd851a8-185e-4b99-bcb6-76c0d1c63f1c\") " pod="openshift-image-registry/node-ca-vhjjf"
Sep 11 21:52:14.827569 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.827016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl55q\" (UniqueName: \"kubernetes.io/projected/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-kube-api-access-cl55q\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.827569 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.827119    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-host\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.827569 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.827178    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fedd4091-aaa3-44e1-9643-2a3e5c30be81-host-slash\") pod \"iptables-alerter-g2swb\" (UID: \"fedd4091-aaa3-44e1-9643-2a3e5c30be81\") " pod="openshift-network-operator/iptables-alerter-g2swb"
Sep 11 21:52:14.827569 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.827227    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-sysconfig\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.827569 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.827244    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-device-dir\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.827569 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.827527    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-var-lib-openvswitch\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.827942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-tuned\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.827998    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-registration-dir\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828080    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-895lc\" (UniqueName: \"kubernetes.io/projected/fbd851a8-185e-4b99-bcb6-76c0d1c63f1c-kube-api-access-895lc\") pod \"node-ca-vhjjf\" (UID: \"fbd851a8-185e-4b99-bcb6-76c0d1c63f1c\") " pod="openshift-image-registry/node-ca-vhjjf"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828173    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-etc-openvswitch\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828227    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-dbus\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-modprobe-d\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-run-ovn\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828343    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-sysctl-d\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828382    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-run\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828415    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fbd851a8-185e-4b99-bcb6-76c0d1c63f1c-host\") pod \"node-ca-vhjjf\" (UID: \"fbd851a8-185e-4b99-bcb6-76c0d1c63f1c\") " pod="openshift-image-registry/node-ca-vhjjf"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828449    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-systemd-units\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828486    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-run-netns\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828507    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828521    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-run-openvswitch\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.828588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828574    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-run-openvswitch\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.829512 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828610    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-kubernetes\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.829512 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-etc-openvswitch\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.829512 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828702    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-dbus\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:14.829512 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828811    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-modprobe-d\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.829512 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.828892    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-run-ovn\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.829512 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829016    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-sysctl-d\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.829512 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829088    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-run\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.829512 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829146    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fbd851a8-185e-4b99-bcb6-76c0d1c63f1c-host\") pod \"node-ca-vhjjf\" (UID: \"fbd851a8-185e-4b99-bcb6-76c0d1c63f1c\") " pod="openshift-image-registry/node-ca-vhjjf"
Sep 11 21:52:14.829512 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829194    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-systemd-units\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.829512 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829238    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-run-netns\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.829512 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829345    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-ovnkube-script-lib\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.829512 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829515    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-registration-dir\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.830111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-kubernetes\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.830111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-systemd\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.830111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829695    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-ovnkube-config\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.830111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829726    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-sysctl-conf\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.830111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829804    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.830111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829839    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae03deae-80f3-4af0-b22c-edd9123df3ff-tmp\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.830111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829852    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv922\" (UniqueName: \"kubernetes.io/projected/f1c80315-39cc-4397-be35-3d7de7f7e9a1-kube-api-access-bv922\") pod \"node-resolver-js4ll\" (UID: \"f1c80315-39cc-4397-be35-3d7de7f7e9a1\") " pod="openshift-dns/node-resolver-js4ll"
Sep 11 21:52:14.830111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-run-systemd\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.830111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829914    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-env-overrides\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.830111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.829954    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-systemd\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.830111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830028    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fedd4091-aaa3-44e1-9643-2a3e5c30be81-iptables-alerter-script\") pod \"iptables-alerter-g2swb\" (UID: \"fedd4091-aaa3-44e1-9643-2a3e5c30be81\") " pod="openshift-network-operator/iptables-alerter-g2swb"
Sep 11 21:52:14.830111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830064    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c80315-39cc-4397-be35-3d7de7f7e9a1-tmp-dir\") pod \"node-resolver-js4ll\" (UID: \"f1c80315-39cc-4397-be35-3d7de7f7e9a1\") " pod="openshift-dns/node-resolver-js4ll"
Sep 11 21:52:14.830111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72dxs\" (UniqueName: \"kubernetes.io/projected/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-kube-api-access-72dxs\") pod \"network-metrics-daemon-9swth\" (UID: \"f07461d1-42d3-43f6-bf6a-89a8663f4ca8\") " pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:14.830742 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-cni-bin\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.830742 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830218    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-kubelet-config\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:14.830742 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830268    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39564d9a-8271-4a01-bfe2-8e7c30c227b2-konnectivity-ca\") pod \"konnectivity-agent-52qz9\" (UID: \"39564d9a-8271-4a01-bfe2-8e7c30c227b2\") " pod="kube-system/konnectivity-agent-52qz9"
Sep 11 21:52:14.830742 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830301    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-slash\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.830742 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830394    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-sys\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.830742 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830520    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-ovnkube-config\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.830742 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830566    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fbd851a8-185e-4b99-bcb6-76c0d1c63f1c-serviceca\") pod \"node-ca-vhjjf\" (UID: \"fbd851a8-185e-4b99-bcb6-76c0d1c63f1c\") " pod="openshift-image-registry/node-ca-vhjjf"
Sep 11 21:52:14.830742 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830589    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.830742 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sgx9\" (UniqueName: \"kubernetes.io/projected/fedd4091-aaa3-44e1-9643-2a3e5c30be81-kube-api-access-2sgx9\") pod \"iptables-alerter-g2swb\" (UID: \"fedd4091-aaa3-44e1-9643-2a3e5c30be81\") " pod="openshift-network-operator/iptables-alerter-g2swb"
Sep 11 21:52:14.831163 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830765    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-run-systemd\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.831163 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830887    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-sysctl-conf\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.831163 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830944    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-sys-fs\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.831163 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.830989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhg9w\" (UniqueName: \"kubernetes.io/projected/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-kube-api-access-bhg9w\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.831163 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831024    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs\") pod \"network-metrics-daemon-9swth\" (UID: \"f07461d1-42d3-43f6-bf6a-89a8663f4ca8\") " pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:14.831163 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831060    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-lib-modules\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.831163 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831091    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-var-lib-kubelet\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.831163 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831124    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-socket-dir\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.831163 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831150    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-node-log\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.831607 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-cni-netd\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.831607 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831239    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49kb\" (UniqueName: \"kubernetes.io/projected/ae03deae-80f3-4af0-b22c-edd9123df3ff-kube-api-access-j49kb\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.831607 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1c80315-39cc-4397-be35-3d7de7f7e9a1-hosts-file\") pod \"node-resolver-js4ll\" (UID: \"f1c80315-39cc-4397-be35-3d7de7f7e9a1\") " pod="openshift-dns/node-resolver-js4ll"
Sep 11 21:52:14.831607 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-log-socket\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.831607 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831543    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-env-overrides\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.831607 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831591    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-log-socket\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.831881 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831683    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-ovn-node-metrics-cert\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.831881 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831847    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae03deae-80f3-4af0-b22c-edd9123df3ff-etc-tuned\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.831970 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831922    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-cni-bin\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.832021 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.831969    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-sys-fs\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.832139    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fedd4091-aaa3-44e1-9643-2a3e5c30be81-iptables-alerter-script\") pod \"iptables-alerter-g2swb\" (UID: \"fedd4091-aaa3-44e1-9643-2a3e5c30be81\") " pod="openshift-network-operator/iptables-alerter-g2swb"
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.832203    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c80315-39cc-4397-be35-3d7de7f7e9a1-tmp-dir\") pod \"node-resolver-js4ll\" (UID: \"f1c80315-39cc-4397-be35-3d7de7f7e9a1\") " pod="openshift-dns/node-resolver-js4ll"
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.832292    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-lib-modules\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.832365    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-kubelet-config\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.832414    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.832440    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-slash\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.832495    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs podName:f07461d1-42d3-43f6-bf6a-89a8663f4ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:15.332477388 +0000 UTC m=+2.067588732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs") pod "network-metrics-daemon-9swth" (UID: "f07461d1-42d3-43f6-bf6a-89a8663f4ca8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.832538    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1c80315-39cc-4397-be35-3d7de7f7e9a1-hosts-file\") pod \"node-resolver-js4ll\" (UID: \"f1c80315-39cc-4397-be35-3d7de7f7e9a1\") " pod="openshift-dns/node-resolver-js4ll"
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.832570    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-sys\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.832627    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-host-cni-netd\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.832666    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39564d9a-8271-4a01-bfe2-8e7c30c227b2-konnectivity-ca\") pod \"konnectivity-agent-52qz9\" (UID: \"39564d9a-8271-4a01-bfe2-8e7c30c227b2\") " pod="kube-system/konnectivity-agent-52qz9"
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.832679    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-node-log\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.832703    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-socket-dir\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.833795 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.832762    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae03deae-80f3-4af0-b22c-edd9123df3ff-var-lib-kubelet\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.834433 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.834416    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39564d9a-8271-4a01-bfe2-8e7c30c227b2-agent-certs\") pod \"konnectivity-agent-52qz9\" (UID: \"39564d9a-8271-4a01-bfe2-8e7c30c227b2\") " pod="kube-system/konnectivity-agent-52qz9"
Sep 11 21:52:14.835493 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.835480    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:14.835541 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.835497    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:14.835541 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.835507    2574 projected.go:196] Error preparing data for projected volume kube-api-access-jftm7 for pod openshift-network-diagnostics/network-check-target-5h829: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:14.835605 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:14.835550    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7 podName:ec958662-21bf-4b7a-89e8-267265fec58f nodeName:}" failed. No retries permitted until 2026-09-11 21:52:15.335539191 +0000 UTC m=+2.070650521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jftm7" (UniqueName: "kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7") pod "network-check-target-5h829" (UID: "ec958662-21bf-4b7a-89e8-267265fec58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:14.837917 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.837890    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-895lc\" (UniqueName: \"kubernetes.io/projected/fbd851a8-185e-4b99-bcb6-76c0d1c63f1c-kube-api-access-895lc\") pod \"node-ca-vhjjf\" (UID: \"fbd851a8-185e-4b99-bcb6-76c0d1c63f1c\") " pod="openshift-image-registry/node-ca-vhjjf"
Sep 11 21:52:14.838008 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.837929    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl55q\" (UniqueName: \"kubernetes.io/projected/f0cf56f5-b9bb-4eba-8d83-eca6c7178e13-kube-api-access-cl55q\") pod \"ovnkube-node-wzl98\" (UID: \"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.839122 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.839097    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sgx9\" (UniqueName: \"kubernetes.io/projected/fedd4091-aaa3-44e1-9643-2a3e5c30be81-kube-api-access-2sgx9\") pod \"iptables-alerter-g2swb\" (UID: \"fedd4091-aaa3-44e1-9643-2a3e5c30be81\") " pod="openshift-network-operator/iptables-alerter-g2swb"
Sep 11 21:52:14.839729 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.839709    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv922\" (UniqueName: \"kubernetes.io/projected/f1c80315-39cc-4397-be35-3d7de7f7e9a1-kube-api-access-bv922\") pod \"node-resolver-js4ll\" (UID: \"f1c80315-39cc-4397-be35-3d7de7f7e9a1\") " pod="openshift-dns/node-resolver-js4ll"
Sep 11 21:52:14.840089 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.840065    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72dxs\" (UniqueName: \"kubernetes.io/projected/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-kube-api-access-72dxs\") pod \"network-metrics-daemon-9swth\" (UID: \"f07461d1-42d3-43f6-bf6a-89a8663f4ca8\") " pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:14.840303 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.840287    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhg9w\" (UniqueName: \"kubernetes.io/projected/1ef1f24d-d62e-43f7-ac50-b572ed5067ed-kube-api-access-bhg9w\") pod \"aws-ebs-csi-driver-node-58shb\" (UID: \"1ef1f24d-d62e-43f7-ac50-b572ed5067ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:14.840831 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.840815    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49kb\" (UniqueName: \"kubernetes.io/projected/ae03deae-80f3-4af0-b22c-edd9123df3ff-kube-api-access-j49kb\") pod \"tuned-wzp59\" (UID: \"ae03deae-80f3-4af0-b22c-edd9123df3ff\") " pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:14.941868 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.941819    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-frptb"
Sep 11 21:52:14.951814 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:14.951792    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc735de58_8252_4bfe_b52d_5b1682fe92b8.slice/crio-7f27d3956e04fef098aa843ffab9afd5fd4af3a3c1a5d1d13fa0d0657bdf6bea WatchSource:0}: Error finding container 7f27d3956e04fef098aa843ffab9afd5fd4af3a3c1a5d1d13fa0d0657bdf6bea: Status 404 returned error can't find the container with id 7f27d3956e04fef098aa843ffab9afd5fd4af3a3c1a5d1d13fa0d0657bdf6bea
Sep 11 21:52:14.963491 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.963469    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qcn7v"
Sep 11 21:52:14.971334 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.971312    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g2swb"
Sep 11 21:52:14.971604 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:14.971584    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6071a312_2b91_4da6_aaf0_993a791e109f.slice/crio-0f32701cab082323317c916e675f6bf259389839645e3d286df278efa5d83069 WatchSource:0}: Error finding container 0f32701cab082323317c916e675f6bf259389839645e3d286df278efa5d83069: Status 404 returned error can't find the container with id 0f32701cab082323317c916e675f6bf259389839645e3d286df278efa5d83069
Sep 11 21:52:14.979038 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:14.979017    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfedd4091_aaa3_44e1_9643_2a3e5c30be81.slice/crio-37bc79846d062d98f036471117a1ce9b20a8e7cfe8b72a870866c4395255d755 WatchSource:0}: Error finding container 37bc79846d062d98f036471117a1ce9b20a8e7cfe8b72a870866c4395255d755: Status 404 returned error can't find the container with id 37bc79846d062d98f036471117a1ce9b20a8e7cfe8b72a870866c4395255d755
Sep 11 21:52:14.984379 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.984361    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:14.992736 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:14.992718    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0cf56f5_b9bb_4eba_8d83_eca6c7178e13.slice/crio-6ca53376012f7e002b838bf3ef2bca6cd635fe7c81b44243cfdab8959b7644fa WatchSource:0}: Error finding container 6ca53376012f7e002b838bf3ef2bca6cd635fe7c81b44243cfdab8959b7644fa: Status 404 returned error can't find the container with id 6ca53376012f7e002b838bf3ef2bca6cd635fe7c81b44243cfdab8959b7644fa
Sep 11 21:52:14.999459 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:14.999438    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb"
Sep 11 21:52:15.008067 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:15.008045    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef1f24d_d62e_43f7_ac50_b572ed5067ed.slice/crio-596c5b905d7435729885f9b6db9c929f2a2dc38d11e17c4143b4e72c881dab41 WatchSource:0}: Error finding container 596c5b905d7435729885f9b6db9c929f2a2dc38d11e17c4143b4e72c881dab41: Status 404 returned error can't find the container with id 596c5b905d7435729885f9b6db9c929f2a2dc38d11e17c4143b4e72c881dab41
Sep 11 21:52:15.017175 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.017157    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-js4ll"
Sep 11 21:52:15.025358 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:15.025326    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c80315_39cc_4397_be35_3d7de7f7e9a1.slice/crio-0b68f6c611c517c87b7d508527b166f5705423dc3f909b92c89929d0ab0a2d22 WatchSource:0}: Error finding container 0b68f6c611c517c87b7d508527b166f5705423dc3f909b92c89929d0ab0a2d22: Status 404 returned error can't find the container with id 0b68f6c611c517c87b7d508527b166f5705423dc3f909b92c89929d0ab0a2d22
Sep 11 21:52:15.030380 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.030363    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-52qz9"
Sep 11 21:52:15.038417 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:15.038385    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39564d9a_8271_4a01_bfe2_8e7c30c227b2.slice/crio-6141d2da204f8b8f43d5538df95f43bae8dee240841eb5a66dca719696f4f4fd WatchSource:0}: Error finding container 6141d2da204f8b8f43d5538df95f43bae8dee240841eb5a66dca719696f4f4fd: Status 404 returned error can't find the container with id 6141d2da204f8b8f43d5538df95f43bae8dee240841eb5a66dca719696f4f4fd
Sep 11 21:52:15.038817 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.038800    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:15.049404 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.049383    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wzp59"
Sep 11 21:52:15.054339 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.054305    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vhjjf"
Sep 11 21:52:15.059316 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:15.059297    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae03deae_80f3_4af0_b22c_edd9123df3ff.slice/crio-01aec3d34757fb3413d46e320d2149a3ed0be3e3c1ab9f91a2d1f2b5b2cf32b6 WatchSource:0}: Error finding container 01aec3d34757fb3413d46e320d2149a3ed0be3e3c1ab9f91a2d1f2b5b2cf32b6: Status 404 returned error can't find the container with id 01aec3d34757fb3413d46e320d2149a3ed0be3e3c1ab9f91a2d1f2b5b2cf32b6
Sep 11 21:52:15.065031 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:15.064700    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd851a8_185e_4b99_bcb6_76c0d1c63f1c.slice/crio-833763dc7f896584f92ea4a2d8cc1521f17cc5fda806cff81f750a676b517908 WatchSource:0}: Error finding container 833763dc7f896584f92ea4a2d8cc1521f17cc5fda806cff81f750a676b517908: Status 404 returned error can't find the container with id 833763dc7f896584f92ea4a2d8cc1521f17cc5fda806cff81f750a676b517908
Sep 11 21:52:15.335429 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.335318    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs\") pod \"network-metrics-daemon-9swth\" (UID: \"f07461d1-42d3-43f6-bf6a-89a8663f4ca8\") " pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:15.335429 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.335377    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:15.335646 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:15.335484    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:15.335646 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:15.335491    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:15.335646 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:15.335541    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret podName:b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa nodeName:}" failed. No retries permitted until 2026-09-11 21:52:16.335525637 +0000 UTC m=+3.070636968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret") pod "global-pull-secret-syncer-628px" (UID: "b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:15.335646 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:15.335567    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs podName:f07461d1-42d3-43f6-bf6a-89a8663f4ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:16.335559508 +0000 UTC m=+3.070670834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs") pod "network-metrics-daemon-9swth" (UID: "f07461d1-42d3-43f6-bf6a-89a8663f4ca8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:15.436375 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.436324    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftm7\" (UniqueName: \"kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7\") pod \"network-check-target-5h829\" (UID: \"ec958662-21bf-4b7a-89e8-267265fec58f\") " pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:15.436574 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:15.436507    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:15.436574 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:15.436529    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:15.436574 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:15.436543    2574 projected.go:196] Error preparing data for projected volume kube-api-access-jftm7 for pod openshift-network-diagnostics/network-check-target-5h829: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:15.436763 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:15.436606    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7 podName:ec958662-21bf-4b7a-89e8-267265fec58f nodeName:}" failed. No retries permitted until 2026-09-11 21:52:16.436587049 +0000 UTC m=+3.171698390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jftm7" (UniqueName: "kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7") pod "network-check-target-5h829" (UID: "ec958662-21bf-4b7a-89e8-267265fec58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:15.667250 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.667135    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:47:15 +0000 UTC" deadline="2028-04-26 20:46:35.023763374 +0000 UTC"
Sep 11 21:52:15.667250 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.667187    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14230h54m19.356581789s"
Sep 11 21:52:15.746330 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.746236    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wzp59" event={"ID":"ae03deae-80f3-4af0-b22c-edd9123df3ff","Type":"ContainerStarted","Data":"01aec3d34757fb3413d46e320d2149a3ed0be3e3c1ab9f91a2d1f2b5b2cf32b6"}
Sep 11 21:52:15.752080 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.752044    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb" event={"ID":"1ef1f24d-d62e-43f7-ac50-b572ed5067ed","Type":"ContainerStarted","Data":"596c5b905d7435729885f9b6db9c929f2a2dc38d11e17c4143b4e72c881dab41"}
Sep 11 21:52:15.763310 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.763281    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g2swb" event={"ID":"fedd4091-aaa3-44e1-9643-2a3e5c30be81","Type":"ContainerStarted","Data":"37bc79846d062d98f036471117a1ce9b20a8e7cfe8b72a870866c4395255d755"}
Sep 11 21:52:15.783855 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.783823    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qcn7v" event={"ID":"6071a312-2b91-4da6-aaf0-993a791e109f","Type":"ContainerStarted","Data":"0f32701cab082323317c916e675f6bf259389839645e3d286df278efa5d83069"}
Sep 11 21:52:15.801757 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.801725    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" event={"ID":"ba1918520a962a77d1f14b1726c5bd1b","Type":"ContainerStarted","Data":"3679f7168f017511dd8a41463987cd8272e7ae3ed9bfdbcddd2a566dfe950990"}
Sep 11 21:52:15.835593 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.835563    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-52qz9" event={"ID":"39564d9a-8271-4a01-bfe2-8e7c30c227b2","Type":"ContainerStarted","Data":"6141d2da204f8b8f43d5538df95f43bae8dee240841eb5a66dca719696f4f4fd"}
Sep 11 21:52:15.853136 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.853107    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-js4ll" event={"ID":"f1c80315-39cc-4397-be35-3d7de7f7e9a1","Type":"ContainerStarted","Data":"0b68f6c611c517c87b7d508527b166f5705423dc3f909b92c89929d0ab0a2d22"}
Sep 11 21:52:15.870718 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.870683    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98" event={"ID":"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13","Type":"ContainerStarted","Data":"6ca53376012f7e002b838bf3ef2bca6cd635fe7c81b44243cfdab8959b7644fa"}
Sep 11 21:52:15.878078 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.878052    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frptb" event={"ID":"c735de58-8252-4bfe-b52d-5b1682fe92b8","Type":"ContainerStarted","Data":"7f27d3956e04fef098aa843ffab9afd5fd4af3a3c1a5d1d13fa0d0657bdf6bea"}
Sep 11 21:52:15.887499 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.887477    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:15.898927 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.898902    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal" event={"ID":"91e28972004d554b545a91ff30beb546","Type":"ContainerStarted","Data":"201a2b716ea7ca981308b606a7e88e44c7679cb0a837067cc67d38f8cdf2f443"}
Sep 11 21:52:15.929361 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:15.929296    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vhjjf" event={"ID":"fbd851a8-185e-4b99-bcb6-76c0d1c63f1c","Type":"ContainerStarted","Data":"833763dc7f896584f92ea4a2d8cc1521f17cc5fda806cff81f750a676b517908"}
Sep 11 21:52:16.344164 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:16.344060    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs\") pod \"network-metrics-daemon-9swth\" (UID: \"f07461d1-42d3-43f6-bf6a-89a8663f4ca8\") " pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:16.344164 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:16.344135    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:16.344448 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:16.344295    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:16.344448 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:16.344375    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret podName:b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa nodeName:}" failed. No retries permitted until 2026-09-11 21:52:18.344338262 +0000 UTC m=+5.079449594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret") pod "global-pull-secret-syncer-628px" (UID: "b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:16.344793 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:16.344772    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:16.344876 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:16.344829    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs podName:f07461d1-42d3-43f6-bf6a-89a8663f4ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:18.344813742 +0000 UTC m=+5.079925075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs") pod "network-metrics-daemon-9swth" (UID: "f07461d1-42d3-43f6-bf6a-89a8663f4ca8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:16.445849 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:16.445809    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftm7\" (UniqueName: \"kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7\") pod \"network-check-target-5h829\" (UID: \"ec958662-21bf-4b7a-89e8-267265fec58f\") " pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:16.446033 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:16.446022    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:16.446096 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:16.446042    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:16.446096 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:16.446058    2574 projected.go:196] Error preparing data for projected volume kube-api-access-jftm7 for pod openshift-network-diagnostics/network-check-target-5h829: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:16.446203 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:16.446120    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7 podName:ec958662-21bf-4b7a-89e8-267265fec58f nodeName:}" failed. No retries permitted until 2026-09-11 21:52:18.446101278 +0000 UTC m=+5.181212622 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jftm7" (UniqueName: "kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7") pod "network-check-target-5h829" (UID: "ec958662-21bf-4b7a-89e8-267265fec58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:16.668275 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:16.668152    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:47:15 +0000 UTC" deadline="2028-02-29 01:20:26.479660142 +0000 UTC"
Sep 11 21:52:16.668275 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:16.668194    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12843h28m9.811470887s"
Sep 11 21:52:16.714470 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:16.714254    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:16.714470 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:16.714426    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:16.715403 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:16.714868    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:16.715403 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:16.714975    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:16.715403 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:16.715042    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:16.715403 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:16.715119    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:16.745097 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:16.744538    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:17.408283 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:17.408248    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:18.366717 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:18.365803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs\") pod \"network-metrics-daemon-9swth\" (UID: \"f07461d1-42d3-43f6-bf6a-89a8663f4ca8\") " pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:18.366717 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:18.365869    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:18.366717 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:18.366057    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:18.366717 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:18.366126    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret podName:b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa nodeName:}" failed. No retries permitted until 2026-09-11 21:52:22.366107576 +0000 UTC m=+9.101218906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret") pod "global-pull-secret-syncer-628px" (UID: "b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:18.366717 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:18.366618    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:18.366717 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:18.366675    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs podName:f07461d1-42d3-43f6-bf6a-89a8663f4ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:22.366657825 +0000 UTC m=+9.101769216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs") pod "network-metrics-daemon-9swth" (UID: "f07461d1-42d3-43f6-bf6a-89a8663f4ca8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:18.467143 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:18.467063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftm7\" (UniqueName: \"kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7\") pod \"network-check-target-5h829\" (UID: \"ec958662-21bf-4b7a-89e8-267265fec58f\") " pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:18.467350 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:18.467295    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:18.467350 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:18.467318    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:18.467350 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:18.467333    2574 projected.go:196] Error preparing data for projected volume kube-api-access-jftm7 for pod openshift-network-diagnostics/network-check-target-5h829: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:18.467564 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:18.467410    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7 podName:ec958662-21bf-4b7a-89e8-267265fec58f nodeName:}" failed. No retries permitted until 2026-09-11 21:52:22.467389689 +0000 UTC m=+9.202501032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jftm7" (UniqueName: "kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7") pod "network-check-target-5h829" (UID: "ec958662-21bf-4b7a-89e8-267265fec58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:18.714331 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:18.714287    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:18.714502 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:18.714461    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:18.714859 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:18.714834    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:18.714958 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:18.714834    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:18.714995 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:18.714956    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:18.715062 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:18.715038    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:20.714810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:20.714751    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:20.715189 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:20.714951    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:20.715382 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:20.715339    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:20.715640 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:20.715476    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:20.715640 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:20.715531    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:20.715640 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:20.715601    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:22.403977 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:22.403941    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs\") pod \"network-metrics-daemon-9swth\" (UID: \"f07461d1-42d3-43f6-bf6a-89a8663f4ca8\") " pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:22.404433 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:22.403998    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:22.404433 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:22.404135    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:22.404433 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:22.404143    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:22.404433 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:22.404200    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret podName:b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa nodeName:}" failed. No retries permitted until 2026-09-11 21:52:30.404182335 +0000 UTC m=+17.139293665 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret") pod "global-pull-secret-syncer-628px" (UID: "b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:22.404433 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:22.404220    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs podName:f07461d1-42d3-43f6-bf6a-89a8663f4ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:30.404209773 +0000 UTC m=+17.139321103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs") pod "network-metrics-daemon-9swth" (UID: "f07461d1-42d3-43f6-bf6a-89a8663f4ca8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:22.504768 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:22.504736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftm7\" (UniqueName: \"kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7\") pod \"network-check-target-5h829\" (UID: \"ec958662-21bf-4b7a-89e8-267265fec58f\") " pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:22.504942 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:22.504912    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:22.504942 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:22.504936    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:22.505012 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:22.504950    2574 projected.go:196] Error preparing data for projected volume kube-api-access-jftm7 for pod openshift-network-diagnostics/network-check-target-5h829: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:22.505043 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:22.505014    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7 podName:ec958662-21bf-4b7a-89e8-267265fec58f nodeName:}" failed. No retries permitted until 2026-09-11 21:52:30.504995272 +0000 UTC m=+17.240106603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jftm7" (UniqueName: "kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7") pod "network-check-target-5h829" (UID: "ec958662-21bf-4b7a-89e8-267265fec58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:22.714558 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:22.714478    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:22.714558 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:22.714508    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:22.714774 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:22.714478    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:22.714774 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:22.714604    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:22.714774 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:22.714721    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:22.714926 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:22.714797    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:24.714563 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:24.714480    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:24.715008 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:24.714611    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:24.715069 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:24.715019    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:24.715153 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:24.715122    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:24.715288 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:24.715185    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:24.715288 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:24.715253    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:26.714235 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:26.714208    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:26.714606 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:26.714208    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:26.714606 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:26.714301    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:26.714606 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:26.714208    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:26.714606 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:26.714396    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:26.714606 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:26.714464    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:28.714153 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:28.714121    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:28.714153 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:28.714165    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:28.714665 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:28.714165    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:28.714665 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:28.714255    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:28.714665 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:28.714354    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:28.714665 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:28.714429    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:30.459881 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:30.459852    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs\") pod \"network-metrics-daemon-9swth\" (UID: \"f07461d1-42d3-43f6-bf6a-89a8663f4ca8\") " pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:30.460385 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:30.459898    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:30.460385 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:30.460004    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:30.460385 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:30.460020    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:30.460385 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:30.460053    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret podName:b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa nodeName:}" failed. No retries permitted until 2026-09-11 21:52:46.460039063 +0000 UTC m=+33.195150387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret") pod "global-pull-secret-syncer-628px" (UID: "b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:30.460385 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:30.460084    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs podName:f07461d1-42d3-43f6-bf6a-89a8663f4ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:46.460065423 +0000 UTC m=+33.195176752 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs") pod "network-metrics-daemon-9swth" (UID: "f07461d1-42d3-43f6-bf6a-89a8663f4ca8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:30.561131 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:30.561097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftm7\" (UniqueName: \"kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7\") pod \"network-check-target-5h829\" (UID: \"ec958662-21bf-4b7a-89e8-267265fec58f\") " pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:30.561289 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:30.561241    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:30.561289 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:30.561257    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:30.561289 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:30.561269    2574 projected.go:196] Error preparing data for projected volume kube-api-access-jftm7 for pod openshift-network-diagnostics/network-check-target-5h829: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:30.561476 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:30.561342    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7 podName:ec958662-21bf-4b7a-89e8-267265fec58f nodeName:}" failed. No retries permitted until 2026-09-11 21:52:46.561323522 +0000 UTC m=+33.296434867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jftm7" (UniqueName: "kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7") pod "network-check-target-5h829" (UID: "ec958662-21bf-4b7a-89e8-267265fec58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:30.713779 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:30.713696    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:30.713779 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:30.713721    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:30.713779 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:30.713738    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:30.714027 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:30.713820    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:30.714027 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:30.713866    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:30.714027 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:30.713898    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:32.714032 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:32.714003    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:32.714609 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:32.714006    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:32.714609 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:32.714111    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:32.714609 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:32.714128    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:32.714609 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:32.714214    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:32.714609 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:32.714273    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:34.713789 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:34.713742    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:34.714190 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:34.713833    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:34.714190 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:34.713839    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:34.714190 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:34.713876    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:34.714190 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:34.713952    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:34.714190 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:34.714028    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:34.985342 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:34.985019    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qcn7v" event={"ID":"6071a312-2b91-4da6-aaf0-993a791e109f","Type":"ContainerStarted","Data":"9588a8c0d05a60d469424800e89edaf72d6cdccaa41f32172d1d0d1cc1c0a355"}
Sep 11 21:52:34.988540 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:34.988455    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 21:52:34.988927 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:34.988900    2574 generic.go:356] "Generic (PLEG): container finished" podID="f0cf56f5-b9bb-4eba-8d83-eca6c7178e13" containerID="26f034794761320e34358ee19fb045242d66ade2d1a83081549526a8ab927d0f" exitCode=1
Sep 11 21:52:34.989024 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:34.988983    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98" event={"ID":"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13","Type":"ContainerStarted","Data":"ef23c4f1a2c048d893dc6d1c1bdea9ea7c9606882ff83acfaf9af64b4e77c27a"}
Sep 11 21:52:34.989089 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:34.989051    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98" event={"ID":"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13","Type":"ContainerDied","Data":"26f034794761320e34358ee19fb045242d66ade2d1a83081549526a8ab927d0f"}
Sep 11 21:52:34.989139 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:34.989088    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98" event={"ID":"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13","Type":"ContainerStarted","Data":"a02b0148bb1bcf49373f3ef1d77c8a8adfae678f40e176e6807da2f0b3ed3b42"}
Sep 11 21:52:34.991213 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:34.990803    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal" event={"ID":"91e28972004d554b545a91ff30beb546","Type":"ContainerStarted","Data":"3b5bbd9ef9dcc82708dba3b184a7e656ed9462672abc847a9c31d91adcbf3ffb"}
Sep 11 21:52:34.997268 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:34.996835    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wzp59" event={"ID":"ae03deae-80f3-4af0-b22c-edd9123df3ff","Type":"ContainerStarted","Data":"6d2025181641cc804b255508f579fd7ccd92fec098fad2350696c66c7732e557"}
Sep 11 21:52:35.023436 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:35.023275    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qcn7v" podStartSLOduration=1.216763106 podStartE2EDuration="21.023254112s" totalImagesPullingTime="19.806491006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:35.006988541 +0000 UTC m=+21.742099889" watchObservedRunningTime="2026-09-11 21:52:35.023254112 +0000 UTC m=+21.758365455"
Sep 11 21:52:35.024397 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:35.023960    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal" podStartSLOduration=20.023947136 podStartE2EDuration="20.023947136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:35.023529353 +0000 UTC m=+21.758640699" watchObservedRunningTime="2026-09-11 21:52:35.023947136 +0000 UTC m=+21.759058487"
Sep 11 21:52:35.041593 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:35.041550    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wzp59" podStartSLOduration=1.5168407529999999 podStartE2EDuration="21.041536489s" totalImagesPullingTime="19.524695736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:35.041415366 +0000 UTC m=+21.776526725" watchObservedRunningTime="2026-09-11 21:52:35.041536489 +0000 UTC m=+21.776647865"
Sep 11 21:52:36.000067 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:35.999828    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vhjjf" event={"ID":"fbd851a8-185e-4b99-bcb6-76c0d1c63f1c","Type":"ContainerStarted","Data":"5646b1247bf77faf196f312e6898fd7c19ffecc4ca6548a7d852f05a61e886e5"}
Sep 11 21:52:36.001405 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.001382    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb" event={"ID":"1ef1f24d-d62e-43f7-ac50-b572ed5067ed","Type":"ContainerStarted","Data":"31c7c8b6be3a50ada05f14cd0e01d62ca11412e8b2ef7273b064694c06819367"}
Sep 11 21:52:36.002842 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.002821    2574 generic.go:356] "Generic (PLEG): container finished" podID="ba1918520a962a77d1f14b1726c5bd1b" containerID="008122887ac57ba0694af70e09d321f6d8ac956864c8246c90641eb1bbfaae54" exitCode=0
Sep 11 21:52:36.002930 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.002882    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" event={"ID":"ba1918520a962a77d1f14b1726c5bd1b","Type":"ContainerDied","Data":"008122887ac57ba0694af70e09d321f6d8ac956864c8246c90641eb1bbfaae54"}
Sep 11 21:52:36.004358 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.004333    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-52qz9" event={"ID":"39564d9a-8271-4a01-bfe2-8e7c30c227b2","Type":"ContainerStarted","Data":"9c61db310a6888efba2c519d41b361a882304012560fe0099ec4e46c9f0253d6"}
Sep 11 21:52:36.005841 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.005821    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-js4ll" event={"ID":"f1c80315-39cc-4397-be35-3d7de7f7e9a1","Type":"ContainerStarted","Data":"ef34b95868842cdaf85f679bfbced7a8c307812c8f112cc1344883cc54f6e3d4"}
Sep 11 21:52:36.008521 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.008503    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 21:52:36.008888 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.008864    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98" event={"ID":"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13","Type":"ContainerStarted","Data":"1c4c783549305c08e39a630d2e155cbdb7c23f95b27e6b93f8203c15b7ed7f5e"}
Sep 11 21:52:36.008972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.008938    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98" event={"ID":"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13","Type":"ContainerStarted","Data":"21f4818bf7ca9ed541f4fb72306a31bfff6964b3b358a40c39fb4170c1f25543"}
Sep 11 21:52:36.009011 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.008986    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98" event={"ID":"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13","Type":"ContainerStarted","Data":"f2476fe6db40c1193af9b806c9e70105d4a888ca6900b68805ccefc2dfc91e16"}
Sep 11 21:52:36.010248 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.010229    2574 generic.go:356] "Generic (PLEG): container finished" podID="c735de58-8252-4bfe-b52d-5b1682fe92b8" containerID="aed09e0c5a824e58def8b790983b74ca83ad51cff83b36884436b3e3fc6f2b7f" exitCode=0
Sep 11 21:52:36.010351 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.010289    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frptb" event={"ID":"c735de58-8252-4bfe-b52d-5b1682fe92b8","Type":"ContainerDied","Data":"aed09e0c5a824e58def8b790983b74ca83ad51cff83b36884436b3e3fc6f2b7f"}
Sep 11 21:52:36.014400 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.014356    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vhjjf" podStartSLOduration=2.506305465 podStartE2EDuration="22.014339527s" totalImagesPullingTime="19.508034062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:36.014260723 +0000 UTC m=+22.749372070" watchObservedRunningTime="2026-09-11 21:52:36.014339527 +0000 UTC m=+22.749450876"
Sep 11 21:52:36.047069 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.047028    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-52qz9" podStartSLOduration=2.525670501 podStartE2EDuration="22.047016164s" totalImagesPullingTime="19.521345663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:36.046877765 +0000 UTC m=+22.781989111" watchObservedRunningTime="2026-09-11 21:52:36.047016164 +0000 UTC m=+22.782127509"
Sep 11 21:52:36.083580 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.083534    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-js4ll" podStartSLOduration=2.5250289280000002 podStartE2EDuration="22.083522063s" totalImagesPullingTime="19.558493135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:36.083173252 +0000 UTC m=+22.818284598" watchObservedRunningTime="2026-09-11 21:52:36.083522063 +0000 UTC m=+22.818633409"
Sep 11 21:52:36.537658 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.537638    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 21:52:36.688939 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.688844    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T21:52:36.537654834Z","UUID":"2af9af26-9b74-4be7-ac60-75b6c1939022","Handler":null,"Name":"","Endpoint":""}
Sep 11 21:52:36.690670 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.690649    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 21:52:36.690752 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.690680    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 21:52:36.714426 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.714401    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:36.714534 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.714428    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:36.714534 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:36.714437    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:36.714534 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:36.714501    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:36.714696 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:36.714632    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:36.714734 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:36.714713    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:37.015783 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:37.015708    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb" event={"ID":"1ef1f24d-d62e-43f7-ac50-b572ed5067ed","Type":"ContainerStarted","Data":"d0c418653f7b5b6331a6eb35d4d6af21db6a21be583d219f7bc0b259938fd5bc"}
Sep 11 21:52:37.017283 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:37.017259    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g2swb" event={"ID":"fedd4091-aaa3-44e1-9643-2a3e5c30be81","Type":"ContainerStarted","Data":"c0732a9f9fe989605ba2bac438d973be82bae1c28bfdb5c18c0715a86e81c1b3"}
Sep 11 21:52:37.019383 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:37.019358    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" event={"ID":"ba1918520a962a77d1f14b1726c5bd1b","Type":"ContainerStarted","Data":"a4546e05461d1e6ad5eef6e688e01f23f5b9746867bceda2e45d8671d0da45e9"}
Sep 11 21:52:37.032475 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:37.032423    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g2swb" podStartSLOduration=3.438424243 podStartE2EDuration="23.032407484s" totalImagesPullingTime="19.593983241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:37.032059475 +0000 UTC m=+23.767170813" watchObservedRunningTime="2026-09-11 21:52:37.032407484 +0000 UTC m=+23.767518831"
Sep 11 21:52:37.047470 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:37.047430    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" podStartSLOduration=22.047418842 podStartE2EDuration="22.047418842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:37.047035749 +0000 UTC m=+23.782147094" watchObservedRunningTime="2026-09-11 21:52:37.047418842 +0000 UTC m=+23.782530207"
Sep 11 21:52:38.025695 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:38.025672    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 21:52:38.026331 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:38.026280    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98" event={"ID":"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13","Type":"ContainerStarted","Data":"5d2d74280f8b70472cd328541fec18bed6607ac081fd5a4afec35a06b79fe324"}
Sep 11 21:52:38.713876 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:38.713671    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:38.714035 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:38.713677    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:38.714035 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:38.713960    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:38.714168 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:38.714055    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:38.714168 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:38.713722    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:38.714265 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:38.714172    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:39.031238 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:39.031156    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb" event={"ID":"1ef1f24d-d62e-43f7-ac50-b572ed5067ed","Type":"ContainerStarted","Data":"a756bea0cc1327c7ea052f833f359d0c472e8c979a5491abb4897eb41265100c"}
Sep 11 21:52:39.049888 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:39.049837    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-58shb" podStartSLOduration=9.999139121 podStartE2EDuration="25.049822093s" totalImagesPullingTime="15.050682972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:39.049444559 +0000 UTC m=+25.784556022" watchObservedRunningTime="2026-09-11 21:52:39.049822093 +0000 UTC m=+25.784933439"
Sep 11 21:52:40.421154 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:40.421120    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-52qz9"
Sep 11 21:52:40.421800 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:40.421780    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-52qz9"
Sep 11 21:52:40.714295 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:40.714222    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:40.714467 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:40.714336    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:40.714467 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:40.714347    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:40.714467 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:40.714361    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:40.714467 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:40.714435    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:40.714661 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:40.714506    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:41.039911 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:41.039845    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 21:52:41.040907 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:41.040450    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98" event={"ID":"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13","Type":"ContainerStarted","Data":"94574a7952f2cbb0ad764b3ee9c9090163d41131048de0191e9d92341b26ebda"}
Sep 11 21:52:41.040907 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:41.040807    2574 scope.go:117] "RemoveContainer" containerID="26f034794761320e34358ee19fb045242d66ade2d1a83081549526a8ab927d0f"
Sep 11 21:52:42.046923 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.046707    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 21:52:42.047429 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.047282    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98" event={"ID":"f0cf56f5-b9bb-4eba-8d83-eca6c7178e13","Type":"ContainerStarted","Data":"10cf2fda8ea250707c4829b45bfbf7536110cf9906f7f361471378c457488124"}
Sep 11 21:52:42.047497 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.047431    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:42.047497 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.047487    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:42.047588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.047533    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:42.053272 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.053237    2574 generic.go:356] "Generic (PLEG): container finished" podID="c735de58-8252-4bfe-b52d-5b1682fe92b8" containerID="ee9f7d884c7c231450d3ee71b6ec482cc744a766e8a4450f3e0e6533ceb6683f" exitCode=0
Sep 11 21:52:42.053449 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.053285    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frptb" event={"ID":"c735de58-8252-4bfe-b52d-5b1682fe92b8","Type":"ContainerDied","Data":"ee9f7d884c7c231450d3ee71b6ec482cc744a766e8a4450f3e0e6533ceb6683f"}
Sep 11 21:52:42.063532 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.063512    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:42.065113 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.065091    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:52:42.078851 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.078794    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98" podStartSLOduration=8.465431518 podStartE2EDuration="28.078779272s" totalImagesPullingTime="19.613347754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:42.077943679 +0000 UTC m=+28.813055026" watchObservedRunningTime="2026-09-11 21:52:42.078779272 +0000 UTC m=+28.813890618"
Sep 11 21:52:42.714757 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.714718    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:42.714930 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.714760    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:42.714930 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:42.714842    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:42.714930 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:42.714876    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:42.714930 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.714926    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:42.715081 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:42.715009    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:42.784998 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.784957    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-628px"]
Sep 11 21:52:42.788469 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.788436    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9swth"]
Sep 11 21:52:42.789111 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:42.789084    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5h829"]
Sep 11 21:52:43.057008 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:43.056978    2574 generic.go:356] "Generic (PLEG): container finished" podID="c735de58-8252-4bfe-b52d-5b1682fe92b8" containerID="cd557b2db007ed20a286e697064a26ba534f94db004f90ac62f5b61563497d1e" exitCode=0
Sep 11 21:52:43.057553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:43.057045    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:43.057553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:43.057074    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frptb" event={"ID":"c735de58-8252-4bfe-b52d-5b1682fe92b8","Type":"ContainerDied","Data":"cd557b2db007ed20a286e697064a26ba534f94db004f90ac62f5b61563497d1e"}
Sep 11 21:52:43.057553 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:43.057129    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:43.057553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:43.057222    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:43.057553 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:43.057329    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:43.057553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:43.057342    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:43.057553 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:43.057404    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:44.062454 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:44.062235    2574 generic.go:356] "Generic (PLEG): container finished" podID="c735de58-8252-4bfe-b52d-5b1682fe92b8" containerID="3ed37f0da630f7b675dea9ba36fd08faf0f5f7a67189b73a43c8f3abe3983264" exitCode=0
Sep 11 21:52:44.062454 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:44.062327    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frptb" event={"ID":"c735de58-8252-4bfe-b52d-5b1682fe92b8","Type":"ContainerDied","Data":"3ed37f0da630f7b675dea9ba36fd08faf0f5f7a67189b73a43c8f3abe3983264"}
Sep 11 21:52:44.713917 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:44.713892    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:44.714079 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:44.713938    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:44.714079 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:44.713949    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:44.714079 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:44.714025    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:44.714204 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:44.714101    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:44.714204 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:44.714154    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:44.896547 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:44.896076    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-52qz9"
Sep 11 21:52:44.896547 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:44.896201    2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 21:52:44.896901 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:44.896876    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-52qz9"
Sep 11 21:52:46.472552 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:46.472516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:46.472954 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:46.472603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs\") pod \"network-metrics-daemon-9swth\" (UID: \"f07461d1-42d3-43f6-bf6a-89a8663f4ca8\") " pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:46.472954 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:46.472702    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:46.472954 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:46.472747    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs podName:f07461d1-42d3-43f6-bf6a-89a8663f4ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:18.472733578 +0000 UTC m=+65.207844903 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs") pod "network-metrics-daemon-9swth" (UID: "f07461d1-42d3-43f6-bf6a-89a8663f4ca8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:46.473103 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:46.473077    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:46.473241 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:46.473135    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret podName:b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa nodeName:}" failed. No retries permitted until 2026-09-11 21:53:18.473120969 +0000 UTC m=+65.208232294 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret") pod "global-pull-secret-syncer-628px" (UID: "b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:46.573569 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:46.573534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftm7\" (UniqueName: \"kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7\") pod \"network-check-target-5h829\" (UID: \"ec958662-21bf-4b7a-89e8-267265fec58f\") " pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:46.573739 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:46.573686    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:46.573739 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:46.573709    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:46.573739 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:46.573721    2574 projected.go:196] Error preparing data for projected volume kube-api-access-jftm7 for pod openshift-network-diagnostics/network-check-target-5h829: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:46.573897 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:46.573771    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7 podName:ec958662-21bf-4b7a-89e8-267265fec58f nodeName:}" failed. No retries permitted until 2026-09-11 21:53:18.57375871 +0000 UTC m=+65.308870039 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jftm7" (UniqueName: "kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7") pod "network-check-target-5h829" (UID: "ec958662-21bf-4b7a-89e8-267265fec58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:46.714342 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:46.714173    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:46.714342 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:46.714279    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:46.714342 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:46.714293    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628px" podUID="b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa"
Sep 11 21:52:46.714342 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:46.714329    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:46.714672 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:46.714401    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9swth" podUID="f07461d1-42d3-43f6-bf6a-89a8663f4ca8"
Sep 11 21:52:46.714672 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:46.714487    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h829" podUID="ec958662-21bf-4b7a-89e8-267265fec58f"
Sep 11 21:52:47.612879 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.612854    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeReady"
Sep 11 21:52:47.613367 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.612989    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 21:52:47.646812 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.646780    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"]
Sep 11 21:52:47.649770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.649746    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:52:47.652442 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.652415    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56d9f5f6cd-crns2"]
Sep 11 21:52:47.653003 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.652970    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qtd28\""
Sep 11 21:52:47.653003 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.652994    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 11 21:52:47.654985 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.654963    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 21:52:47.655096 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.655080    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 21:52:47.655541 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.655177    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-ckcpn"]
Sep 11 21:52:47.655541 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.655178    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 11 21:52:47.655541 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.655254    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.657919 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.657901    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-889c88f7b-28lxw"]
Sep 11 21:52:47.658064 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.658044    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.658421 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.658362    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 21:52:47.658695 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.658632    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 21:52:47.659425 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.659047    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m5hpf\""
Sep 11 21:52:47.659425 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.659355    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 21:52:47.660851 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.660833    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-lvts5"]
Sep 11 21:52:47.660969 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.660954    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.661818 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.661761    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 21:52:47.661818 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.661787    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 21:52:47.661963 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.661880    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 11 21:52:47.662783 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.662763    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 11 21:52:47.663057 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.663036    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8jw8k\""
Sep 11 21:52:47.663771 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.663753    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8g8rn"]
Sep 11 21:52:47.663914 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.663899    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-lvts5"
Sep 11 21:52:47.665104 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.665084    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-m59xn\""
Sep 11 21:52:47.665202 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.665188    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 11 21:52:47.666926 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.666904    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"]
Sep 11 21:52:47.667036 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.666982    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 11 21:52:47.667204 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.667184    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 21:52:47.667540 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.667385    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 11 21:52:47.667662 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.667624    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:47.667662 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.667634    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 21:52:47.667780 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.667696    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 11 21:52:47.674480 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.674447    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 21:52:47.678166 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.678016    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 21:52:47.678961 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.678940    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:52:47.679525 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.679248    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fmx88\""
Sep 11 21:52:47.679525 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.679426    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 11 21:52:47.680143 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.680117    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 11 21:52:47.682962 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.682942    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"]
Sep 11 21:52:47.683068 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.682989    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-lvts5"]
Sep 11 21:52:47.683068 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.683010    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8g8rn"]
Sep 11 21:52:47.683068 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.682943    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 11 21:52:47.683337 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.683302    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:52:47.683450 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.683431    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-2vfxh\""
Sep 11 21:52:47.683660 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.683644    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:52:47.685539 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.685515    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56d9f5f6cd-crns2"]
Sep 11 21:52:47.685982 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.685961    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 11 21:52:47.686613 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.686589    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k89mx"]
Sep 11 21:52:47.686707 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.686633    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 21:52:47.686876 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.686845    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:52:47.687344 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.687060    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 21:52:47.687344 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.687093    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 11 21:52:47.687344 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.687149    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h6hkz\""
Sep 11 21:52:47.689653 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.689635    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hjw6g"]
Sep 11 21:52:47.689746 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.689729    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k89mx"
Sep 11 21:52:47.694243 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.693848    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qlxmr"]
Sep 11 21:52:47.694243 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.694217    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hjw6g"
Sep 11 21:52:47.696334 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.696261    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 21:52:47.696334 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.696332    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8rpn9\""
Sep 11 21:52:47.696726 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.696681    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 21:52:47.696874 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.696699    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 21:52:47.698042 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.698025    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"]
Sep 11 21:52:47.698534 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.698512    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-m9kmh\""
Sep 11 21:52:47.698890 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.698869    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 21:52:47.699142 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.699112    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 21:52:47.699247 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.698140    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:47.701106 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.701086    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-889c88f7b-28lxw"]
Sep 11 21:52:47.701250 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.701178    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"]
Sep 11 21:52:47.701468 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.701453    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"
Sep 11 21:52:47.701848 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.701826    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8zqrn\""
Sep 11 21:52:47.702130 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.702113    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 21:52:47.702473 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.702451    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 21:52:47.704239 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.704222    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-ckcpn"]
Sep 11 21:52:47.704511 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.704496    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"]
Sep 11 21:52:47.704608 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.704549    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hjw6g"]
Sep 11 21:52:47.704608 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.704571    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"]
Sep 11 21:52:47.704608 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.704588    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qlxmr"]
Sep 11 21:52:47.704608 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.704365    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"
Sep 11 21:52:47.704608 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.704610    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k89mx"]
Sep 11 21:52:47.704895 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.704626    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"]
Sep 11 21:52:47.704895 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.704805    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 11 21:52:47.705075 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.705044    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 11 21:52:47.705266 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.705248    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gx9kl\""
Sep 11 21:52:47.705406 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.705339    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:52:47.705406 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.705383    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 11 21:52:47.707363 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.707343    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 11 21:52:47.707453 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.707390    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:52:47.707645 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.707627    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 11 21:52:47.708160 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.707958    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 11 21:52:47.708160 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.708018    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rc7z2\""
Sep 11 21:52:47.783823 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.783722    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6d54fe9c-b07b-4b7f-96ae-b0af3e131114-config\") pod \"console-operator-696884fb9c-8g8rn\" (UID: \"6d54fe9c-b07b-4b7f-96ae-b0af3e131114\") " pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:47.783823 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.783757    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfxj7\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-kube-api-access-tfxj7\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.783823 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.783800    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd4xm\" (UniqueName: \"kubernetes.io/projected/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-kube-api-access-qd4xm\") pod \"cluster-samples-operator-86499c89c4-2qrmq\" (UID: \"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:52:47.784090 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.783825    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6d54fe9c-b07b-4b7f-96ae-b0af3e131114-serving-cert\") pod \"console-operator-696884fb9c-8g8rn\" (UID: \"6d54fe9c-b07b-4b7f-96ae-b0af3e131114\") " pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:47.784090 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.783853    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4d96ddd-a046-422c-928b-564b04193b6a-installation-pull-secrets\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.784090 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.783878    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99s4\" (UniqueName: \"kubernetes.io/projected/51337e48-d86f-4001-a58a-c6a2297fe84a-kube-api-access-q99s4\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:47.784090 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.783916    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rk5t\" (UniqueName: \"kubernetes.io/projected/6d54fe9c-b07b-4b7f-96ae-b0af3e131114-kube-api-access-4rk5t\") pod \"console-operator-696884fb9c-8g8rn\" (UID: \"6d54fe9c-b07b-4b7f-96ae-b0af3e131114\") " pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:47.784090 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.783962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85b6eeb6-8682-432f-8a62-1a9f9f9ad358-serving-cert\") pod \"service-ca-operator-cf7c776b7-j4dfr\" (UID: \"85b6eeb6-8682-432f-8a62-1a9f9f9ad358\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"
Sep 11 21:52:47.784090 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784020    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51337e48-d86f-4001-a58a-c6a2297fe84a-config-volume\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:47.784090 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784056    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/85b6eeb6-8682-432f-8a62-1a9f9f9ad358-config\") pod \"service-ca-operator-cf7c776b7-j4dfr\" (UID: \"85b6eeb6-8682-432f-8a62-1a9f9f9ad358\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"
Sep 11 21:52:47.784090 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4d96ddd-a046-422c-928b-564b04193b6a-image-registry-private-configuration\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784118    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4d96ddd-a046-422c-928b-564b04193b6a-trusted-ca\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784155    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784186    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsfj\" (UniqueName: \"kubernetes.io/projected/1d26ad8d-4464-4bfc-981c-7fde3e3cdf48-kube-api-access-zhsfj\") pod \"volume-data-source-validator-86695b7cc7-lvts5\" (UID: \"1d26ad8d-4464-4bfc-981c-7fde3e3cdf48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-lvts5"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784211    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-stats-auth\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784245    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlqd4\" (UniqueName: \"kubernetes.io/projected/2eecf508-6314-4920-a2c6-4d90d43db253-kube-api-access-jlqd4\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4d96ddd-a046-422c-928b-564b04193b6a-ca-trust-extracted\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784303    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-bound-sa-token\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784354    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784380    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-serving-cert\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784407    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2qrmq\" (UID: \"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbc2g\" (UID: \"90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784457    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7198fa9e-17b2-42b7-b52a-09098330d6a0-cert\") pod \"ingress-canary-k89mx\" (UID: \"7198fa9e-17b2-42b7-b52a-09098330d6a0\") " pod="openshift-ingress-canary/ingress-canary-k89mx"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-service-ca-bundle\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784509    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bgg\" (UniqueName: \"kubernetes.io/projected/4fab439a-fab1-42e3-90bd-7e75c63a1a07-kube-api-access-k8bgg\") pod \"network-check-source-6f8b768455-hjw6g\" (UID: \"4fab439a-fab1-42e3-90bd-7e75c63a1a07\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hjw6g"
Sep 11 21:52:47.784536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784537    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d54fe9c-b07b-4b7f-96ae-b0af3e131114-trusted-ca\") pod \"console-operator-696884fb9c-8g8rn\" (UID: \"6d54fe9c-b07b-4b7f-96ae-b0af3e131114\") " pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784573    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blrdd\" (UniqueName: \"kubernetes.io/projected/7198fa9e-17b2-42b7-b52a-09098330d6a0-kube-api-access-blrdd\") pod \"ingress-canary-k89mx\" (UID: \"7198fa9e-17b2-42b7-b52a-09098330d6a0\") " pod="openshift-ingress-canary/ingress-canary-k89mx"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784640    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbc2g\" (UID: \"90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784664    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhbcx\" (UniqueName: \"kubernetes.io/projected/c667eac2-77fb-4708-a43f-ae76bcac6bc1-kube-api-access-dhbcx\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784689    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n949g\" (UniqueName: \"kubernetes.io/projected/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-kube-api-access-n949g\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784741    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-trusted-ca-bundle\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784782    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz7kb\" (UniqueName: \"kubernetes.io/projected/85b6eeb6-8682-432f-8a62-1a9f9f9ad358-kube-api-access-cz7kb\") pod \"service-ca-operator-cf7c776b7-j4dfr\" (UID: \"85b6eeb6-8682-432f-8a62-1a9f9f9ad358\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4d96ddd-a046-422c-928b-564b04193b6a-registry-certificates\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784833    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-snapshots\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784856    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-default-certificate\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784879    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-tmp\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784907    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ltzn\" (UniqueName: \"kubernetes.io/projected/90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a-kube-api-access-9ltzn\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbc2g\" (UID: \"90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.784956    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2eecf508-6314-4920-a2c6-4d90d43db253-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.785008    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51337e48-d86f-4001-a58a-c6a2297fe84a-tmp-dir\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:47.785290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.785045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.785852 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.785086    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:52:47.820509 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.820482    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:47.885619 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.885570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bgg\" (UniqueName: \"kubernetes.io/projected/4fab439a-fab1-42e3-90bd-7e75c63a1a07-kube-api-access-k8bgg\") pod \"network-check-source-6f8b768455-hjw6g\" (UID: \"4fab439a-fab1-42e3-90bd-7e75c63a1a07\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hjw6g"
Sep 11 21:52:47.885770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.885627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d54fe9c-b07b-4b7f-96ae-b0af3e131114-trusted-ca\") pod \"console-operator-696884fb9c-8g8rn\" (UID: \"6d54fe9c-b07b-4b7f-96ae-b0af3e131114\") " pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:47.885770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.885654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blrdd\" (UniqueName: \"kubernetes.io/projected/7198fa9e-17b2-42b7-b52a-09098330d6a0-kube-api-access-blrdd\") pod \"ingress-canary-k89mx\" (UID: \"7198fa9e-17b2-42b7-b52a-09098330d6a0\") " pod="openshift-ingress-canary/ingress-canary-k89mx"
Sep 11 21:52:47.885770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.885682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbc2g\" (UID: \"90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"
Sep 11 21:52:47.885770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.885706    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhbcx\" (UniqueName: \"kubernetes.io/projected/c667eac2-77fb-4708-a43f-ae76bcac6bc1-kube-api-access-dhbcx\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.885987 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.885842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n949g\" (UniqueName: \"kubernetes.io/projected/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-kube-api-access-n949g\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.885987 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.885892    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.885987 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.885924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-trusted-ca-bundle\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.885987 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.885950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz7kb\" (UniqueName: \"kubernetes.io/projected/85b6eeb6-8682-432f-8a62-1a9f9f9ad358-kube-api-access-cz7kb\") pod \"service-ca-operator-cf7c776b7-j4dfr\" (UID: \"85b6eeb6-8682-432f-8a62-1a9f9f9ad358\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"
Sep 11 21:52:47.886143 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.885999    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4d96ddd-a046-422c-928b-564b04193b6a-registry-certificates\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.886143 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886033    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-snapshots\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.886143 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:47.886050    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle podName:c667eac2-77fb-4708-a43f-ae76bcac6bc1 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:48.386030169 +0000 UTC m=+35.121141497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle") pod "router-default-889c88f7b-28lxw" (UID: "c667eac2-77fb-4708-a43f-ae76bcac6bc1") : configmap references non-existent config key: service-ca.crt
Sep 11 21:52:47.886143 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-default-certificate\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.886143 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886120    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-tmp\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.886421 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ltzn\" (UniqueName: \"kubernetes.io/projected/90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a-kube-api-access-9ltzn\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbc2g\" (UID: \"90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"
Sep 11 21:52:47.886421 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886182    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2eecf508-6314-4920-a2c6-4d90d43db253-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:52:47.886421 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886212    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51337e48-d86f-4001-a58a-c6a2297fe84a-tmp-dir\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:47.886421 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886242    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.886421 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886346    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbc2g\" (UID: \"90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"
Sep 11 21:52:47.886657 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886458    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:52:47.886657 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886521    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6d54fe9c-b07b-4b7f-96ae-b0af3e131114-config\") pod \"console-operator-696884fb9c-8g8rn\" (UID: \"6d54fe9c-b07b-4b7f-96ae-b0af3e131114\") " pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:47.886657 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:47.886538    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 21:52:47.886657 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:47.886602    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs podName:c667eac2-77fb-4708-a43f-ae76bcac6bc1 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:48.386584895 +0000 UTC m=+35.121696237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs") pod "router-default-889c88f7b-28lxw" (UID: "c667eac2-77fb-4708-a43f-ae76bcac6bc1") : secret "router-metrics-certs-default" not found
Sep 11 21:52:47.886657 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886604    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d54fe9c-b07b-4b7f-96ae-b0af3e131114-trusted-ca\") pod \"console-operator-696884fb9c-8g8rn\" (UID: \"6d54fe9c-b07b-4b7f-96ae-b0af3e131114\") " pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:47.886891 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886728    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-snapshots\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.886891 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfxj7\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-kube-api-access-tfxj7\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.886891 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886820    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd4xm\" (UniqueName: \"kubernetes.io/projected/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-kube-api-access-qd4xm\") pod \"cluster-samples-operator-86499c89c4-2qrmq\" (UID: \"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:52:47.886891 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886844    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6d54fe9c-b07b-4b7f-96ae-b0af3e131114-serving-cert\") pod \"console-operator-696884fb9c-8g8rn\" (UID: \"6d54fe9c-b07b-4b7f-96ae-b0af3e131114\") " pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:47.886891 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886857    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-tmp\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.886891 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886871    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4d96ddd-a046-422c-928b-564b04193b6a-installation-pull-secrets\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.886891 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886880    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-trusted-ca-bundle\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886897    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99s4\" (UniqueName: \"kubernetes.io/projected/51337e48-d86f-4001-a58a-c6a2297fe84a-kube-api-access-q99s4\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886912    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51337e48-d86f-4001-a58a-c6a2297fe84a-tmp-dir\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886937    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rk5t\" (UniqueName: \"kubernetes.io/projected/6d54fe9c-b07b-4b7f-96ae-b0af3e131114-kube-api-access-4rk5t\") pod \"console-operator-696884fb9c-8g8rn\" (UID: \"6d54fe9c-b07b-4b7f-96ae-b0af3e131114\") " pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85b6eeb6-8682-432f-8a62-1a9f9f9ad358-serving-cert\") pod \"service-ca-operator-cf7c776b7-j4dfr\" (UID: \"85b6eeb6-8682-432f-8a62-1a9f9f9ad358\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.886989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51337e48-d86f-4001-a58a-c6a2297fe84a-config-volume\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887015    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/85b6eeb6-8682-432f-8a62-1a9f9f9ad358-config\") pod \"service-ca-operator-cf7c776b7-j4dfr\" (UID: \"85b6eeb6-8682-432f-8a62-1a9f9f9ad358\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887039    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4d96ddd-a046-422c-928b-564b04193b6a-image-registry-private-configuration\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887058    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4d96ddd-a046-422c-928b-564b04193b6a-registry-certificates\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:47.886968    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:47.887123    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls podName:2eecf508-6314-4920-a2c6-4d90d43db253 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:48.387109373 +0000 UTC m=+35.122220711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-h65fx" (UID: "2eecf508-6314-4920-a2c6-4d90d43db253") : secret "cluster-monitoring-operator-tls" not found
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4d96ddd-a046-422c-928b-564b04193b6a-trusted-ca\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887149    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2eecf508-6314-4920-a2c6-4d90d43db253-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887161    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:47.887223 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887215    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsfj\" (UniqueName: \"kubernetes.io/projected/1d26ad8d-4464-4bfc-981c-7fde3e3cdf48-kube-api-access-zhsfj\") pod \"volume-data-source-validator-86695b7cc7-lvts5\" (UID: \"1d26ad8d-4464-4bfc-981c-7fde3e3cdf48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-lvts5"
Sep 11 21:52:47.887908 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887242    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-stats-auth\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.887908 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887272    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlqd4\" (UniqueName: \"kubernetes.io/projected/2eecf508-6314-4920-a2c6-4d90d43db253-kube-api-access-jlqd4\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:52:47.887908 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887298    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4d96ddd-a046-422c-928b-564b04193b6a-ca-trust-extracted\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.887908 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887342    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-bound-sa-token\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.887908 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.887908 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887396    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-serving-cert\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.887908 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887426    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2qrmq\" (UID: \"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:52:47.887908 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887461    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbc2g\" (UID: \"90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"
Sep 11 21:52:47.887908 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887485    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7198fa9e-17b2-42b7-b52a-09098330d6a0-cert\") pod \"ingress-canary-k89mx\" (UID: \"7198fa9e-17b2-42b7-b52a-09098330d6a0\") " pod="openshift-ingress-canary/ingress-canary-k89mx"
Sep 11 21:52:47.887908 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887519    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-service-ca-bundle\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.887908 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887709    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51337e48-d86f-4001-a58a-c6a2297fe84a-config-volume\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:47.888453 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.887984    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4d96ddd-a046-422c-928b-564b04193b6a-trusted-ca\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.888453 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.888025    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-service-ca-bundle\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.888453 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.888092    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6d54fe9c-b07b-4b7f-96ae-b0af3e131114-config\") pod \"console-operator-696884fb9c-8g8rn\" (UID: \"6d54fe9c-b07b-4b7f-96ae-b0af3e131114\") " pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:47.888453 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.888204    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/85b6eeb6-8682-432f-8a62-1a9f9f9ad358-config\") pod \"service-ca-operator-cf7c776b7-j4dfr\" (UID: \"85b6eeb6-8682-432f-8a62-1a9f9f9ad358\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"
Sep 11 21:52:47.888453 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:47.888240    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:52:47.888453 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:47.888255    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56d9f5f6cd-crns2: secret "image-registry-tls" not found
Sep 11 21:52:47.888453 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:47.888290    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls podName:c4d96ddd-a046-422c-928b-564b04193b6a nodeName:}" failed. No retries permitted until 2026-09-11 21:52:48.388278665 +0000 UTC m=+35.123390008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls") pod "image-registry-56d9f5f6cd-crns2" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a") : secret "image-registry-tls" not found
Sep 11 21:52:47.888861 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:47.888520    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 21:52:47.888861 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:47.888556    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls podName:1a452c74-ccd8-4cbf-8850-d9f0bf78ac50 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:48.388544589 +0000 UTC m=+35.123655916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2qrmq" (UID: "1a452c74-ccd8-4cbf-8850-d9f0bf78ac50") : secret "samples-operator-tls" not found
Sep 11 21:52:47.888861 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.888589    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4d96ddd-a046-422c-928b-564b04193b6a-ca-trust-extracted\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.889079 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:47.888950    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:52:47.889079 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:47.889011    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls podName:51337e48-d86f-4001-a58a-c6a2297fe84a nodeName:}" failed. No retries permitted until 2026-09-11 21:52:48.388995113 +0000 UTC m=+35.124106437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls") pod "dns-default-qlxmr" (UID: "51337e48-d86f-4001-a58a-c6a2297fe84a") : secret "dns-default-metrics-tls" not found
Sep 11 21:52:47.892129 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.892103    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85b6eeb6-8682-432f-8a62-1a9f9f9ad358-serving-cert\") pod \"service-ca-operator-cf7c776b7-j4dfr\" (UID: \"85b6eeb6-8682-432f-8a62-1a9f9f9ad358\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"
Sep 11 21:52:47.892253 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.892152    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6d54fe9c-b07b-4b7f-96ae-b0af3e131114-serving-cert\") pod \"console-operator-696884fb9c-8g8rn\" (UID: \"6d54fe9c-b07b-4b7f-96ae-b0af3e131114\") " pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:47.892349 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.892331    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-stats-auth\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.892585 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.892560    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-default-certificate\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.892687 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.892639    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7198fa9e-17b2-42b7-b52a-09098330d6a0-cert\") pod \"ingress-canary-k89mx\" (UID: \"7198fa9e-17b2-42b7-b52a-09098330d6a0\") " pod="openshift-ingress-canary/ingress-canary-k89mx"
Sep 11 21:52:47.894189 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.893433    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4d96ddd-a046-422c-928b-564b04193b6a-image-registry-private-configuration\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.894189 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.893527    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-serving-cert\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.894189 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.893642    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbc2g\" (UID: \"90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"
Sep 11 21:52:47.895047 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.894952    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4d96ddd-a046-422c-928b-564b04193b6a-installation-pull-secrets\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.896692 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.896069    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bgg\" (UniqueName: \"kubernetes.io/projected/4fab439a-fab1-42e3-90bd-7e75c63a1a07-kube-api-access-k8bgg\") pod \"network-check-source-6f8b768455-hjw6g\" (UID: \"4fab439a-fab1-42e3-90bd-7e75c63a1a07\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hjw6g"
Sep 11 21:52:47.896692 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.896570    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz7kb\" (UniqueName: \"kubernetes.io/projected/85b6eeb6-8682-432f-8a62-1a9f9f9ad358-kube-api-access-cz7kb\") pod \"service-ca-operator-cf7c776b7-j4dfr\" (UID: \"85b6eeb6-8682-432f-8a62-1a9f9f9ad358\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"
Sep 11 21:52:47.896850 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.896776    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blrdd\" (UniqueName: \"kubernetes.io/projected/7198fa9e-17b2-42b7-b52a-09098330d6a0-kube-api-access-blrdd\") pod \"ingress-canary-k89mx\" (UID: \"7198fa9e-17b2-42b7-b52a-09098330d6a0\") " pod="openshift-ingress-canary/ingress-canary-k89mx"
Sep 11 21:52:47.897069 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.897048    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhbcx\" (UniqueName: \"kubernetes.io/projected/c667eac2-77fb-4708-a43f-ae76bcac6bc1-kube-api-access-dhbcx\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:47.899456 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.898874    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfxj7\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-kube-api-access-tfxj7\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.899456 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.898916    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n949g\" (UniqueName: \"kubernetes.io/projected/64b6b508-ccca-4b2f-8e17-44de2a0b6e55-kube-api-access-n949g\") pod \"insights-operator-5844b856db-ckcpn\" (UID: \"64b6b508-ccca-4b2f-8e17-44de2a0b6e55\") " pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:47.899456 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.899418    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd4xm\" (UniqueName: \"kubernetes.io/projected/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-kube-api-access-qd4xm\") pod \"cluster-samples-operator-86499c89c4-2qrmq\" (UID: \"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:52:47.899672 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.899567    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-bound-sa-token\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:47.899726 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.899694    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsfj\" (UniqueName: \"kubernetes.io/projected/1d26ad8d-4464-4bfc-981c-7fde3e3cdf48-kube-api-access-zhsfj\") pod \"volume-data-source-validator-86695b7cc7-lvts5\" (UID: \"1d26ad8d-4464-4bfc-981c-7fde3e3cdf48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-lvts5"
Sep 11 21:52:47.899909 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.899886    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ltzn\" (UniqueName: \"kubernetes.io/projected/90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a-kube-api-access-9ltzn\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbc2g\" (UID: \"90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"
Sep 11 21:52:47.900032 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.900013    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlqd4\" (UniqueName: \"kubernetes.io/projected/2eecf508-6314-4920-a2c6-4d90d43db253-kube-api-access-jlqd4\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:52:47.901074 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.901049    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99s4\" (UniqueName: \"kubernetes.io/projected/51337e48-d86f-4001-a58a-c6a2297fe84a-kube-api-access-q99s4\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:47.902261 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.902232    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rk5t\" (UniqueName: \"kubernetes.io/projected/6d54fe9c-b07b-4b7f-96ae-b0af3e131114-kube-api-access-4rk5t\") pod \"console-operator-696884fb9c-8g8rn\" (UID: \"6d54fe9c-b07b-4b7f-96ae-b0af3e131114\") " pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:47.909733 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.909707    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:47.988371 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:47.988346    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-ckcpn"
Sep 11 21:52:48.009124 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.009102    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-lvts5"
Sep 11 21:52:48.017948 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.017929    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:52:48.030735 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.030712    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k89mx"
Sep 11 21:52:48.037427 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.037367    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hjw6g"
Sep 11 21:52:48.050146 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.050126    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"
Sep 11 21:52:48.055798 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.055782    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"
Sep 11 21:52:48.391593 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.391560    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:48.391756 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.391599    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:52:48.391756 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:48.391711    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 21:52:48.391756 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:48.391728    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 21:52:48.391881 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:48.391769    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls podName:2eecf508-6314-4920-a2c6-4d90d43db253 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:49.391749764 +0000 UTC m=+36.126861101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-h65fx" (UID: "2eecf508-6314-4920-a2c6-4d90d43db253") : secret "cluster-monitoring-operator-tls" not found
Sep 11 21:52:48.391881 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:48.391786    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs podName:c667eac2-77fb-4708-a43f-ae76bcac6bc1 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:49.391777788 +0000 UTC m=+36.126889114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs") pod "router-default-889c88f7b-28lxw" (UID: "c667eac2-77fb-4708-a43f-ae76bcac6bc1") : secret "router-metrics-certs-default" not found
Sep 11 21:52:48.391881 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.391839    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:48.391881 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.391880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:48.392064 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.391899    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2qrmq\" (UID: \"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:52:48.392064 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.391950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:48.392064 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:48.391986    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:52:48.392064 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:48.392000    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56d9f5f6cd-crns2: secret "image-registry-tls" not found
Sep 11 21:52:48.392064 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:48.392041    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls podName:c4d96ddd-a046-422c-928b-564b04193b6a nodeName:}" failed. No retries permitted until 2026-09-11 21:52:49.39202905 +0000 UTC m=+36.127140391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls") pod "image-registry-56d9f5f6cd-crns2" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a") : secret "image-registry-tls" not found
Sep 11 21:52:48.392064 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:48.391986    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:52:48.392393 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:48.392070    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 21:52:48.392393 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:48.392080    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls podName:51337e48-d86f-4001-a58a-c6a2297fe84a nodeName:}" failed. No retries permitted until 2026-09-11 21:52:49.392068437 +0000 UTC m=+36.127179765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls") pod "dns-default-qlxmr" (UID: "51337e48-d86f-4001-a58a-c6a2297fe84a") : secret "dns-default-metrics-tls" not found
Sep 11 21:52:48.392393 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:48.392155    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle podName:c667eac2-77fb-4708-a43f-ae76bcac6bc1 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:49.392138381 +0000 UTC m=+36.127249709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle") pod "router-default-889c88f7b-28lxw" (UID: "c667eac2-77fb-4708-a43f-ae76bcac6bc1") : configmap references non-existent config key: service-ca.crt
Sep 11 21:52:48.392393 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:48.392182    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls podName:1a452c74-ccd8-4cbf-8850-d9f0bf78ac50 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:49.392172134 +0000 UTC m=+36.127283460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2qrmq" (UID: "1a452c74-ccd8-4cbf-8850-d9f0bf78ac50") : secret "samples-operator-tls" not found
Sep 11 21:52:48.572479 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.572435    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:48.636663 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.636590    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:48.693616 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.693543    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:48.714709 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.714679    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:52:48.714860 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.714679    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:52:48.714860 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.714679    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:52:48.719803 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.719770    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:52:48.719803 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.719799    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qsk4f\""
Sep 11 21:52:48.719995 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.719802    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4bfmc\""
Sep 11 21:52:48.719995 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:48.719855    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:52:49.347896 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.347870    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k89mx"]
Sep 11 21:52:49.351613 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.351078    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g"]
Sep 11 21:52:49.352140 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:49.352105    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7198fa9e_17b2_42b7_b52a_09098330d6a0.slice/crio-f135191e3ecfe902b00514e956ecf03c61a425ca1fc7e5a795984e080625378d WatchSource:0}: Error finding container f135191e3ecfe902b00514e956ecf03c61a425ca1fc7e5a795984e080625378d: Status 404 returned error can't find the container with id f135191e3ecfe902b00514e956ecf03c61a425ca1fc7e5a795984e080625378d
Sep 11 21:52:49.352616 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.352486    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hjw6g"]
Sep 11 21:52:49.354817 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.354755    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-lvts5"]
Sep 11 21:52:49.354962 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.354936    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-ckcpn"]
Sep 11 21:52:49.358127 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.357441    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr"]
Sep 11 21:52:49.358127 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.357831    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8g8rn"]
Sep 11 21:52:49.363562 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:49.363540    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b6eeb6_8682_432f_8a62_1a9f9f9ad358.slice/crio-4d39e7f4b637928f46990e58202dcb46f37742ba60bde2e97fc6c977b235aab8 WatchSource:0}: Error finding container 4d39e7f4b637928f46990e58202dcb46f37742ba60bde2e97fc6c977b235aab8: Status 404 returned error can't find the container with id 4d39e7f4b637928f46990e58202dcb46f37742ba60bde2e97fc6c977b235aab8
Sep 11 21:52:49.364181 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:49.364084    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d26ad8d_4464_4bfc_981c_7fde3e3cdf48.slice/crio-692a5fd27de2ba4dddca6b888be26a1e80966956eba89d9212d4995d5a029362 WatchSource:0}: Error finding container 692a5fd27de2ba4dddca6b888be26a1e80966956eba89d9212d4995d5a029362: Status 404 returned error can't find the container with id 692a5fd27de2ba4dddca6b888be26a1e80966956eba89d9212d4995d5a029362
Sep 11 21:52:49.364490 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:52:49.364466    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d54fe9c_b07b_4b7f_96ae_b0af3e131114.slice/crio-96de95d3d89824a9dba0d45c855e341059952e5ce182f5e39564d4fc9a0f7430 WatchSource:0}: Error finding container 96de95d3d89824a9dba0d45c855e341059952e5ce182f5e39564d4fc9a0f7430: Status 404 returned error can't find the container with id 96de95d3d89824a9dba0d45c855e341059952e5ce182f5e39564d4fc9a0f7430
Sep 11 21:52:49.400347 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.400192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:49.400442 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:49.400365    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle podName:c667eac2-77fb-4708-a43f-ae76bcac6bc1 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:51.400347712 +0000 UTC m=+38.135459041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle") pod "router-default-889c88f7b-28lxw" (UID: "c667eac2-77fb-4708-a43f-ae76bcac6bc1") : configmap references non-existent config key: service-ca.crt
Sep 11 21:52:49.400442 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.400392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:49.400442 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.400415    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:52:49.400623 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.400477    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:49.400623 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:49.400487    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 21:52:49.400623 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.400502    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:49.400623 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.400520    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2qrmq\" (UID: \"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:52:49.400623 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:49.400537    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs podName:c667eac2-77fb-4708-a43f-ae76bcac6bc1 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:51.400521357 +0000 UTC m=+38.135632701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs") pod "router-default-889c88f7b-28lxw" (UID: "c667eac2-77fb-4708-a43f-ae76bcac6bc1") : secret "router-metrics-certs-default" not found
Sep 11 21:52:49.400623 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:49.400542    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 21:52:49.400623 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:49.400578    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 21:52:49.400623 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:49.400581    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls podName:2eecf508-6314-4920-a2c6-4d90d43db253 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:51.400569802 +0000 UTC m=+38.135681141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-h65fx" (UID: "2eecf508-6314-4920-a2c6-4d90d43db253") : secret "cluster-monitoring-operator-tls" not found
Sep 11 21:52:49.400623 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:49.400580    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:52:49.400623 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:49.400615    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls podName:51337e48-d86f-4001-a58a-c6a2297fe84a nodeName:}" failed. No retries permitted until 2026-09-11 21:52:51.400606585 +0000 UTC m=+38.135717923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls") pod "dns-default-qlxmr" (UID: "51337e48-d86f-4001-a58a-c6a2297fe84a") : secret "dns-default-metrics-tls" not found
Sep 11 21:52:49.400623 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:49.400629    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls podName:1a452c74-ccd8-4cbf-8850-d9f0bf78ac50 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:51.400621286 +0000 UTC m=+38.135732613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2qrmq" (UID: "1a452c74-ccd8-4cbf-8850-d9f0bf78ac50") : secret "samples-operator-tls" not found
Sep 11 21:52:49.401162 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:49.400641    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:52:49.401162 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:49.400654    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56d9f5f6cd-crns2: secret "image-registry-tls" not found
Sep 11 21:52:49.401162 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:49.400688    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls podName:c4d96ddd-a046-422c-928b-564b04193b6a nodeName:}" failed. No retries permitted until 2026-09-11 21:52:51.400679324 +0000 UTC m=+38.135790667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls") pod "image-registry-56d9f5f6cd-crns2" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a") : secret "image-registry-tls" not found
Sep 11 21:52:49.592035 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.592009    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:49.649199 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:49.649172    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:50.079426 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:50.079387    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-lvts5" event={"ID":"1d26ad8d-4464-4bfc-981c-7fde3e3cdf48","Type":"ContainerStarted","Data":"692a5fd27de2ba4dddca6b888be26a1e80966956eba89d9212d4995d5a029362"}
Sep 11 21:52:50.081798 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:50.081750    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g" event={"ID":"90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a","Type":"ContainerStarted","Data":"c1d04e38bc3a9f5992d4d2031fcc48153669af876eda19f7be15e6fd1f6963cb"}
Sep 11 21:52:50.083743 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:50.083675    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k89mx" event={"ID":"7198fa9e-17b2-42b7-b52a-09098330d6a0","Type":"ContainerStarted","Data":"f135191e3ecfe902b00514e956ecf03c61a425ca1fc7e5a795984e080625378d"}
Sep 11 21:52:50.089609 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:50.089545    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr" event={"ID":"85b6eeb6-8682-432f-8a62-1a9f9f9ad358","Type":"ContainerStarted","Data":"4d39e7f4b637928f46990e58202dcb46f37742ba60bde2e97fc6c977b235aab8"}
Sep 11 21:52:50.100918 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:50.100033    2574 generic.go:356] "Generic (PLEG): container finished" podID="c735de58-8252-4bfe-b52d-5b1682fe92b8" containerID="55a05e3614f19d70dfa0c9272f4e5ba404ab8dffdbcece8a50dc0588a08ac1d8" exitCode=0
Sep 11 21:52:50.100918 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:50.100109    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frptb" event={"ID":"c735de58-8252-4bfe-b52d-5b1682fe92b8","Type":"ContainerDied","Data":"55a05e3614f19d70dfa0c9272f4e5ba404ab8dffdbcece8a50dc0588a08ac1d8"}
Sep 11 21:52:50.104396 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:50.104063    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-ckcpn" event={"ID":"64b6b508-ccca-4b2f-8e17-44de2a0b6e55","Type":"ContainerStarted","Data":"15a7a61c683099a2f235255523180492a517ffd30f5df1cfc60b74923b1b260a"}
Sep 11 21:52:50.109775 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:50.109705    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hjw6g" event={"ID":"4fab439a-fab1-42e3-90bd-7e75c63a1a07","Type":"ContainerStarted","Data":"7cdc8a4861d07cd136fe3d154699e02afd9af513080fb7856988033510374825"}
Sep 11 21:52:50.111198 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:50.111129    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn" event={"ID":"6d54fe9c-b07b-4b7f-96ae-b0af3e131114","Type":"ContainerStarted","Data":"96de95d3d89824a9dba0d45c855e341059952e5ce182f5e39564d4fc9a0f7430"}
Sep 11 21:52:51.126707 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:51.126668    2574 generic.go:356] "Generic (PLEG): container finished" podID="c735de58-8252-4bfe-b52d-5b1682fe92b8" containerID="4277050f3e3a4f2b8a857a04e44ac50279e76d832e8fafd90ad0aa790850774a" exitCode=0
Sep 11 21:52:51.127384 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:51.126731    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frptb" event={"ID":"c735de58-8252-4bfe-b52d-5b1682fe92b8","Type":"ContainerDied","Data":"4277050f3e3a4f2b8a857a04e44ac50279e76d832e8fafd90ad0aa790850774a"}
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:51.422065    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:51.422112    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:51.422197    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:51.422237    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:51.422332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2qrmq\" (UID: \"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:51.422388    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:51.422554    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle podName:c667eac2-77fb-4708-a43f-ae76bcac6bc1 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:55.422533105 +0000 UTC m=+42.157644453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle") pod "router-default-889c88f7b-28lxw" (UID: "c667eac2-77fb-4708-a43f-ae76bcac6bc1") : configmap references non-existent config key: service-ca.crt
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:51.422958    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:51.423012    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs podName:c667eac2-77fb-4708-a43f-ae76bcac6bc1 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:55.422996236 +0000 UTC m=+42.158107565 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs") pod "router-default-889c88f7b-28lxw" (UID: "c667eac2-77fb-4708-a43f-ae76bcac6bc1") : secret "router-metrics-certs-default" not found
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:51.423069    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:51.423101    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls podName:2eecf508-6314-4920-a2c6-4d90d43db253 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:55.423090461 +0000 UTC m=+42.158201807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-h65fx" (UID: "2eecf508-6314-4920-a2c6-4d90d43db253") : secret "cluster-monitoring-operator-tls" not found
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:51.423157    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:51.423188    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls podName:51337e48-d86f-4001-a58a-c6a2297fe84a nodeName:}" failed. No retries permitted until 2026-09-11 21:52:55.423177263 +0000 UTC m=+42.158288593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls") pod "dns-default-qlxmr" (UID: "51337e48-d86f-4001-a58a-c6a2297fe84a") : secret "dns-default-metrics-tls" not found
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:51.423250    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:52:51.423417 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:51.423260    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56d9f5f6cd-crns2: secret "image-registry-tls" not found
Sep 11 21:52:51.424204 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:51.423287    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls podName:c4d96ddd-a046-422c-928b-564b04193b6a nodeName:}" failed. No retries permitted until 2026-09-11 21:52:55.423277771 +0000 UTC m=+42.158389107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls") pod "image-registry-56d9f5f6cd-crns2" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a") : secret "image-registry-tls" not found
Sep 11 21:52:51.424204 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:51.423351    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 21:52:51.424204 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:51.423384    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls podName:1a452c74-ccd8-4cbf-8850-d9f0bf78ac50 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:55.423373827 +0000 UTC m=+42.158485159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2qrmq" (UID: "1a452c74-ccd8-4cbf-8850-d9f0bf78ac50") : secret "samples-operator-tls" not found
Sep 11 21:52:51.540396 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:51.540346    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:52.135265 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:52.135223    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frptb" event={"ID":"c735de58-8252-4bfe-b52d-5b1682fe92b8","Type":"ContainerStarted","Data":"8047c214c7d0b81c23689aa0acbacac8ee583c09b7dd0e5eff9b6869049251e3"}
Sep 11 21:52:52.137334 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:52.137255    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-lvts5" event={"ID":"1d26ad8d-4464-4bfc-981c-7fde3e3cdf48","Type":"ContainerStarted","Data":"7a16a139cd06b686329bc2b186559e8cde37c622419c230910c26bdf942838ca"}
Sep 11 21:52:52.159983 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:52.159931    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-frptb" podStartSLOduration=6.740420129 podStartE2EDuration="38.159911768s" totalImagesPullingTime="31.419491639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:52.159086217 +0000 UTC m=+38.894197564" watchObservedRunningTime="2026-09-11 21:52:52.159911768 +0000 UTC m=+38.895023324"
Sep 11 21:52:52.175468 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:52.175422    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-lvts5" podStartSLOduration=21.176638992 podStartE2EDuration="23.175404782s" totalImagesPullingTime="1.99876579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:52.174889691 +0000 UTC m=+38.910001037" watchObservedRunningTime="2026-09-11 21:52:52.175404782 +0000 UTC m=+38.910516144"
Sep 11 21:52:52.281565 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:52.281530    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:52.340610 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:52.340567    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:52.839960 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:52.839877    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:52.940584 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:52.940553    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:53.303065 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:53.302985    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:53.404543 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:53.404448    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:53.997864 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:53.997755    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:54.055683 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:54.055648    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:55.459849 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:55.459814    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:55.459862    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:55.459885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:55.459949    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:55.459978    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle podName:c667eac2-77fb-4708-a43f-ae76bcac6bc1 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:03.459958878 +0000 UTC m=+50.195070230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle") pod "router-default-889c88f7b-28lxw" (UID: "c667eac2-77fb-4708-a43f-ae76bcac6bc1") : configmap references non-existent config key: service-ca.crt
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:55.459989    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:55.460017    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:55.460036    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:55.460047    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs podName:c667eac2-77fb-4708-a43f-ae76bcac6bc1 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:03.460031432 +0000 UTC m=+50.195142776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs") pod "router-default-889c88f7b-28lxw" (UID: "c667eac2-77fb-4708-a43f-ae76bcac6bc1") : secret "router-metrics-certs-default" not found
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:55.460047    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:55.460089    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls podName:51337e48-d86f-4001-a58a-c6a2297fe84a nodeName:}" failed. No retries permitted until 2026-09-11 21:53:03.460082413 +0000 UTC m=+50.195193737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls") pod "dns-default-qlxmr" (UID: "51337e48-d86f-4001-a58a-c6a2297fe84a") : secret "dns-default-metrics-tls" not found
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:55.460089    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:55.460099    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56d9f5f6cd-crns2: secret "image-registry-tls" not found
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:55.460113    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls podName:2eecf508-6314-4920-a2c6-4d90d43db253 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:03.460104519 +0000 UTC m=+50.195215849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-h65fx" (UID: "2eecf508-6314-4920-a2c6-4d90d43db253") : secret "cluster-monitoring-operator-tls" not found
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:55.460128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2qrmq\" (UID: \"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:55.460162    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls podName:c4d96ddd-a046-422c-928b-564b04193b6a nodeName:}" failed. No retries permitted until 2026-09-11 21:53:03.460156847 +0000 UTC m=+50.195268171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls") pod "image-registry-56d9f5f6cd-crns2" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a") : secret "image-registry-tls" not found
Sep 11 21:52:55.460217 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:55.460174    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 21:52:55.460736 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:55.460230    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls podName:1a452c74-ccd8-4cbf-8850-d9f0bf78ac50 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:03.460215743 +0000 UTC m=+50.195327068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2qrmq" (UID: "1a452c74-ccd8-4cbf-8850-d9f0bf78ac50") : secret "samples-operator-tls" not found
Sep 11 21:52:57.153644 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:57.153575    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g" event={"ID":"90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a","Type":"ContainerStarted","Data":"a582f7a7ba9aad1feb7f5fd3471ef20ea6f56f8e3695a6040e8ece65117bb92f"}
Sep 11 21:52:57.156375 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:57.156294    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr" event={"ID":"85b6eeb6-8682-432f-8a62-1a9f9f9ad358","Type":"ContainerStarted","Data":"75eeca2a690f770dc83d648d8e1f1a43fc44d14f0c54e8448adeda65de794c78"}
Sep 11 21:52:57.222617 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:57.220175    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr" podStartSLOduration=15.578573893 podStartE2EDuration="23.220154221s" totalImagesPullingTime="7.641580328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:57.194804217 +0000 UTC m=+43.929915566" watchObservedRunningTime="2026-09-11 21:52:57.220154221 +0000 UTC m=+43.955265570"
Sep 11 21:52:57.222617 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:57.221293    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g" podStartSLOduration=17.574262491 podStartE2EDuration="25.221277231s" totalImagesPullingTime="7.64701474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:57.218480275 +0000 UTC m=+43.953591623" watchObservedRunningTime="2026-09-11 21:52:57.221277231 +0000 UTC m=+43.956388579"
Sep 11 21:52:57.387723 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:57.387673    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:57.487845 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:57.487813    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:58.167105 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.167074    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k89mx" event={"ID":"7198fa9e-17b2-42b7-b52a-09098330d6a0","Type":"ContainerStarted","Data":"e7fef7649495b7006d3466ac8cc67a552b2f3127c8ec0a31d4727e370b493a5a"}
Sep 11 21:52:58.169350 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.169297    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-ckcpn" event={"ID":"64b6b508-ccca-4b2f-8e17-44de2a0b6e55","Type":"ContainerStarted","Data":"dab12bd8b8de2a0ad4e7519ba6887bd54ca57a0c508a49b968414526ad0bebf3"}
Sep 11 21:52:58.170972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.170946    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hjw6g" event={"ID":"4fab439a-fab1-42e3-90bd-7e75c63a1a07","Type":"ContainerStarted","Data":"d2706d37cf07f39916405261951d1fad458c4684e13419be2fd1fcccdb79c5a0"}
Sep 11 21:52:58.173183 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.173161    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/0.log"
Sep 11 21:52:58.173277 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.173233    2574 generic.go:356] "Generic (PLEG): container finished" podID="6d54fe9c-b07b-4b7f-96ae-b0af3e131114" containerID="40329036c52f0418c9051df683b3ce11a2e49368995975cb55904ce5ca212a1a" exitCode=255
Sep 11 21:52:58.173360 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.173304    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn" event={"ID":"6d54fe9c-b07b-4b7f-96ae-b0af3e131114","Type":"ContainerDied","Data":"40329036c52f0418c9051df683b3ce11a2e49368995975cb55904ce5ca212a1a"}
Sep 11 21:52:58.173504 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.173487    2574 scope.go:117] "RemoveContainer" containerID="40329036c52f0418c9051df683b3ce11a2e49368995975cb55904ce5ca212a1a"
Sep 11 21:52:58.184428 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.184353    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k89mx" podStartSLOduration=3.535308511 podStartE2EDuration="11.1843346s" totalImagesPullingTime="7.649026089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:58.183355439 +0000 UTC m=+44.918466789" watchObservedRunningTime="2026-09-11 21:52:58.1843346 +0000 UTC m=+44.919445961"
Sep 11 21:52:58.220232 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.219061    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hjw6g" podStartSLOduration=15.561644661999999 podStartE2EDuration="23.219039965s" totalImagesPullingTime="7.657395303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:58.218382934 +0000 UTC m=+44.953494281" watchObservedRunningTime="2026-09-11 21:52:58.219039965 +0000 UTC m=+44.954151314"
Sep 11 21:52:58.235270 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.235101    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-ckcpn" podStartSLOduration=22.5916532 podStartE2EDuration="30.235080685s" totalImagesPullingTime="7.643427485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:58.234376589 +0000 UTC m=+44.969487935" watchObservedRunningTime="2026-09-11 21:52:58.235080685 +0000 UTC m=+44.970192031"
Sep 11 21:52:58.382632 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.382606    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8gxh6"]
Sep 11 21:52:58.389702 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.389676    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:58.402233 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.402209    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8gxh6"]
Sep 11 21:52:58.402461 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.402436    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:58.408324 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.405496    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k6pq6\""
Sep 11 21:52:58.408324 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.405710    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 21:52:58.408324 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.405917    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 21:52:58.459990 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.459955    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:58.489200 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.489166    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-crio-socket\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:58.489362 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.489253    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:58.489362 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.489342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:58.489460 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.489373    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nftfn\" (UniqueName: \"kubernetes.io/projected/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-api-access-nftfn\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:58.489460 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.489414    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-data-volume\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:58.590116 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.590087    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-data-volume\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:58.590265 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.590163    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-crio-socket\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:58.590265 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.590200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:58.590265 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.590233    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:58.590265 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.590258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nftfn\" (UniqueName: \"kubernetes.io/projected/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-api-access-nftfn\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:58.590483 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:58.590327    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:52:58.590483 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:58.590384    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:52:58.590483 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.590427    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-crio-socket\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:58.590483 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:58.590391    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm podName:d92733da-a0e5-4341-aed9-ad49e2a6b4f5 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:59.0903739 +0000 UTC m=+45.825485241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8gxh6" (UID: "d92733da-a0e5-4341-aed9-ad49e2a6b4f5") : configmap "kube-rbac-proxy" not found
Sep 11 21:52:58.590632 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:58.590486    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls podName:d92733da-a0e5-4341-aed9-ad49e2a6b4f5 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:59.090466955 +0000 UTC m=+45.825578285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8gxh6" (UID: "d92733da-a0e5-4341-aed9-ad49e2a6b4f5") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:52:58.590632 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.590493    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-data-volume\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:58.602305 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:58.602278    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nftfn\" (UniqueName: \"kubernetes.io/projected/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-api-access-nftfn\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:59.021183 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.021149    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:59.093870 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.093835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:59.094042 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.093895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:52:59.094042 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:59.093967    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:52:59.094042 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:59.094036    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm podName:d92733da-a0e5-4341-aed9-ad49e2a6b4f5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:00.094019248 +0000 UTC m=+46.829130577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8gxh6" (UID: "d92733da-a0e5-4341-aed9-ad49e2a6b4f5") : configmap "kube-rbac-proxy" not found
Sep 11 21:52:59.094177 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:59.094056    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:52:59.094177 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:59.094106    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls podName:d92733da-a0e5-4341-aed9-ad49e2a6b4f5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:00.094093712 +0000 UTC m=+46.829205036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8gxh6" (UID: "d92733da-a0e5-4341-aed9-ad49e2a6b4f5") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:52:59.110391 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.110362    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:59.182774 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.182744    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 21:52:59.183267 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.183250    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/0.log"
Sep 11 21:52:59.183340 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.183291    2574 generic.go:356] "Generic (PLEG): container finished" podID="6d54fe9c-b07b-4b7f-96ae-b0af3e131114" containerID="2888bf83cff8de6267b86a78f6c73c058dadfedc9228fdd34e76e80b0ff38a5c" exitCode=255
Sep 11 21:52:59.183417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.183394    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn" event={"ID":"6d54fe9c-b07b-4b7f-96ae-b0af3e131114","Type":"ContainerDied","Data":"2888bf83cff8de6267b86a78f6c73c058dadfedc9228fdd34e76e80b0ff38a5c"}
Sep 11 21:52:59.183494 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.183480    2574 scope.go:117] "RemoveContainer" containerID="40329036c52f0418c9051df683b3ce11a2e49368995975cb55904ce5ca212a1a"
Sep 11 21:52:59.183741 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.183710    2574 scope.go:117] "RemoveContainer" containerID="2888bf83cff8de6267b86a78f6c73c058dadfedc9228fdd34e76e80b0ff38a5c"
Sep 11 21:52:59.183909 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:52:59.183886    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8g8rn_openshift-console-operator(6d54fe9c-b07b-4b7f-96ae-b0af3e131114)\"" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn" podUID="6d54fe9c-b07b-4b7f-96ae-b0af3e131114"
Sep 11 21:52:59.204045 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.204021    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:59.549669 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.549620    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:59.601718 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.601690    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:59.781622 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.781537    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:59.831011 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:52:59.830975    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:00.103610 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:00.103570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:53:00.103784 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:00.103736    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:00.103784 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:00.103755    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:53:00.103909 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:00.103822    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls podName:d92733da-a0e5-4341-aed9-ad49e2a6b4f5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:02.103802198 +0000 UTC m=+48.838913522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8gxh6" (UID: "d92733da-a0e5-4341-aed9-ad49e2a6b4f5") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:00.103909 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:00.103866    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:53:00.104005 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:00.103944    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm podName:d92733da-a0e5-4341-aed9-ad49e2a6b4f5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:02.103926247 +0000 UTC m=+48.839037576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8gxh6" (UID: "d92733da-a0e5-4341-aed9-ad49e2a6b4f5") : configmap "kube-rbac-proxy" not found
Sep 11 21:53:00.189422 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:00.189391    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 21:53:00.189794 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:00.189750    2574 scope.go:117] "RemoveContainer" containerID="2888bf83cff8de6267b86a78f6c73c058dadfedc9228fdd34e76e80b0ff38a5c"
Sep 11 21:53:00.189958 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:00.189938    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8g8rn_openshift-console-operator(6d54fe9c-b07b-4b7f-96ae-b0af3e131114)\"" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn" podUID="6d54fe9c-b07b-4b7f-96ae-b0af3e131114"
Sep 11 21:53:00.254562 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:00.254537    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-js4ll_f1c80315-39cc-4397-be35-3d7de7f7e9a1/dns-node-resolver/0.log"
Sep 11 21:53:00.261812 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:00.261777    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:00.364936 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:00.364855    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:01.251075 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:01.251049    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vhjjf_fbd851a8-185e-4b99-bcb6-76c0d1c63f1c/node-ca/0.log"
Sep 11 21:53:01.851564 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:01.851533    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k89mx_7198fa9e-17b2-42b7-b52a-09098330d6a0/serve-healthcheck-canary/0.log"
Sep 11 21:53:01.875182 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:01.875149    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:01.978631 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:01.978599    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:02.074756 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:02.074728    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:02.122997 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:02.122906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:53:02.122997 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:02.122960    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:53:02.123200 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:02.123022    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:53:02.123200 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:02.123083    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:02.123200 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:02.123087    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm podName:d92733da-a0e5-4341-aed9-ad49e2a6b4f5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:06.123071024 +0000 UTC m=+52.858182348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8gxh6" (UID: "d92733da-a0e5-4341-aed9-ad49e2a6b4f5") : configmap "kube-rbac-proxy" not found
Sep 11 21:53:02.123200 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:02.123138    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls podName:d92733da-a0e5-4341-aed9-ad49e2a6b4f5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:06.123126837 +0000 UTC m=+52.858238164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8gxh6" (UID: "d92733da-a0e5-4341-aed9-ad49e2a6b4f5") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:02.367133 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:02.367101    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:02.468479 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:02.468394    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:02.979526 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:02.979488    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:03.535198 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:03.535159    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:03.535216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:03.535291    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2qrmq\" (UID: \"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:03.535349    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:03.535365    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56d9f5f6cd-crns2: secret "image-registry-tls" not found
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:03.535388    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:03.535427    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls podName:c4d96ddd-a046-422c-928b-564b04193b6a nodeName:}" failed. No retries permitted until 2026-09-11 21:53:19.535408901 +0000 UTC m=+66.270520235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls") pod "image-registry-56d9f5f6cd-crns2" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a") : secret "image-registry-tls" not found
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:03.535437    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:03.535446    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls podName:1a452c74-ccd8-4cbf-8850-d9f0bf78ac50 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:19.535437247 +0000 UTC m=+66.270548571 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2qrmq" (UID: "1a452c74-ccd8-4cbf-8850-d9f0bf78ac50") : secret "samples-operator-tls" not found
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:03.535387    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:03.535462    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle podName:c667eac2-77fb-4708-a43f-ae76bcac6bc1 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:19.535454548 +0000 UTC m=+66.270565878 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle") pod "router-default-889c88f7b-28lxw" (UID: "c667eac2-77fb-4708-a43f-ae76bcac6bc1") : configmap references non-existent config key: service-ca.crt
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:03.535493    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls podName:51337e48-d86f-4001-a58a-c6a2297fe84a nodeName:}" failed. No retries permitted until 2026-09-11 21:53:19.535478282 +0000 UTC m=+66.270589615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls") pod "dns-default-qlxmr" (UID: "51337e48-d86f-4001-a58a-c6a2297fe84a") : secret "dns-default-metrics-tls" not found
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:03.535534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:03.535562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:03.535662    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:03.535674    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 21:53:03.535703 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:03.535708    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs podName:c667eac2-77fb-4708-a43f-ae76bcac6bc1 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:19.535696456 +0000 UTC m=+66.270807795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs") pod "router-default-889c88f7b-28lxw" (UID: "c667eac2-77fb-4708-a43f-ae76bcac6bc1") : secret "router-metrics-certs-default" not found
Sep 11 21:53:03.536214 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:03.535722    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls podName:2eecf508-6314-4920-a2c6-4d90d43db253 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:19.535715881 +0000 UTC m=+66.270827204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-h65fx" (UID: "2eecf508-6314-4920-a2c6-4d90d43db253") : secret "cluster-monitoring-operator-tls" not found
Sep 11 21:53:03.635652 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:03.635616    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:05.430625 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:05.430586    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:05.530951 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:05.530913    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:05.976485 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:05.976457    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:06.080768 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:06.080731    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:06.157264 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:06.157236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:53:06.157427 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:06.157292    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:53:06.157427 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:06.157391    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:06.157520 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:06.157451    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls podName:d92733da-a0e5-4341-aed9-ad49e2a6b4f5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:14.157436755 +0000 UTC m=+60.892548080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8gxh6" (UID: "d92733da-a0e5-4341-aed9-ad49e2a6b4f5") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:06.157821 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:06.157804    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:53:06.178441 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:06.178403    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:07.384055 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:07.384027    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:07.556326 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:07.556261    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:07.657610 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:07.657509    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:07.824185 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:07.824146    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:07.931483 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:07.931394    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:07.985807 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:07.985770    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:08.018363 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:08.018336    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:53:08.018453 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:08.018398    2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:53:08.018748 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:08.018736    2574 scope.go:117] "RemoveContainer" containerID="2888bf83cff8de6267b86a78f6c73c058dadfedc9228fdd34e76e80b0ff38a5c"
Sep 11 21:53:08.018896 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:08.018879    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8g8rn_openshift-console-operator(6d54fe9c-b07b-4b7f-96ae-b0af3e131114)\"" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn" podUID="6d54fe9c-b07b-4b7f-96ae-b0af3e131114"
Sep 11 21:53:08.081093 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:08.081056    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:08.883450 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:08.883412    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:08.993777 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:08.993734    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:09.094916 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:09.094884    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:09.479453 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:09.479422    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:09.684438 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:09.684405    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:10.082276 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:10.082244    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:10.279499 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:10.279471    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:10.684091 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:10.684056    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:11.078699 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:11.078623    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:11.181534 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:11.181503    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:11.281897 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:11.281867    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:11.876969 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:11.876934    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:12.113954 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:12.113920    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:12.165841 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:12.165773    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:12.276238 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:12.276207    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:12.679959 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:12.679923    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:13.285425 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:13.285391    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:13.482776 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:13.482744    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:13.647818 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:13.647788    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:13.749398 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:13.749368    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:14.073229 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:14.073166    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzl98"
Sep 11 21:53:14.226514 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:14.226463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:53:14.228579 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:14.228557    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d92733da-a0e5-4341-aed9-ad49e2a6b4f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gxh6\" (UID: \"d92733da-a0e5-4341-aed9-ad49e2a6b4f5\") " pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:53:14.334330 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:14.334231    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k6pq6\""
Sep 11 21:53:14.341723 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:14.341702    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8gxh6"
Sep 11 21:53:14.552325 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:14.552283    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:14.608078 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:14.608049    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8gxh6"]
Sep 11 21:53:14.611321 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:53:14.611275    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92733da_a0e5_4341_aed9_ad49e2a6b4f5.slice/crio-e7a7bc67657364615d72626fd836d3d52e309ef13d01e44b83343024bdaf754e WatchSource:0}: Error finding container e7a7bc67657364615d72626fd836d3d52e309ef13d01e44b83343024bdaf754e: Status 404 returned error can't find the container with id e7a7bc67657364615d72626fd836d3d52e309ef13d01e44b83343024bdaf754e
Sep 11 21:53:14.885639 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:14.885554    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:15.080182 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:15.080155    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:15.236440 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:15.236369    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8gxh6" event={"ID":"d92733da-a0e5-4341-aed9-ad49e2a6b4f5","Type":"ContainerStarted","Data":"552dab504b6697ab22828cb38134c1f2b65688ef2dcba1a491f6971e166fb235"}
Sep 11 21:53:15.236440 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:15.236431    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8gxh6" event={"ID":"d92733da-a0e5-4341-aed9-ad49e2a6b4f5","Type":"ContainerStarted","Data":"e7a7bc67657364615d72626fd836d3d52e309ef13d01e44b83343024bdaf754e"}
Sep 11 21:53:15.279587 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:15.279555    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:15.477616 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:15.477583    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:15.682289 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:15.682263    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:16.241114 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:16.241078    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8gxh6" event={"ID":"d92733da-a0e5-4341-aed9-ad49e2a6b4f5","Type":"ContainerStarted","Data":"e3f98832242eac6c20cca5a2a577547ea1bf3f3677d4cb8d04858b58c3f62684"}
Sep 11 21:53:16.486882 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:16.486842    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:17.079834 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:17.079754    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:17.245962 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:17.245931    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8gxh6" event={"ID":"d92733da-a0e5-4341-aed9-ad49e2a6b4f5","Type":"ContainerStarted","Data":"d16b5fdc5b7d095775f6471ede55df94e21b5ccfb6ec148f886a1bd20048d1bc"}
Sep 11 21:53:17.268653 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:17.268609    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8gxh6" podStartSLOduration=17.320478751 podStartE2EDuration="19.268595127s" totalImagesPullingTime="1.948116376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:17.265754122 +0000 UTC m=+64.000865468" watchObservedRunningTime="2026-09-11 21:53:17.268595127 +0000 UTC m=+64.003706473"
Sep 11 21:53:17.282054 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:17.282027    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:17.681563 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:17.681534    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:17.877493 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:17.877458    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:18.078656 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.078579    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:18.559587 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.559556    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs\") pod \"network-metrics-daemon-9swth\" (UID: \"f07461d1-42d3-43f6-bf6a-89a8663f4ca8\") " pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:53:18.559750 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.559596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:53:18.562241 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.562222    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:53:18.562300 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.562276    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:53:18.572531 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.572512    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07461d1-42d3-43f6-bf6a-89a8663f4ca8-metrics-certs\") pod \"network-metrics-daemon-9swth\" (UID: \"f07461d1-42d3-43f6-bf6a-89a8663f4ca8\") " pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:53:18.572648 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.572628    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa-original-pull-secret\") pod \"global-pull-secret-syncer-628px\" (UID: \"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa\") " pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:53:18.660433 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.660403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftm7\" (UniqueName: \"kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7\") pod \"network-check-target-5h829\" (UID: \"ec958662-21bf-4b7a-89e8-267265fec58f\") " pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:53:18.662769 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.662748    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jftm7\" (UniqueName: \"kubernetes.io/projected/ec958662-21bf-4b7a-89e8-267265fec58f-kube-api-access-jftm7\") pod \"network-check-target-5h829\" (UID: \"ec958662-21bf-4b7a-89e8-267265fec58f\") " pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:53:18.732195 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.732169    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qsk4f\""
Sep 11 21:53:18.736626 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.736611    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628px"
Sep 11 21:53:18.740635 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.740621    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9swth"
Sep 11 21:53:18.744165 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.744146    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4bfmc\""
Sep 11 21:53:18.752382 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:18.752362    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:53:19.007967 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.007941    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-628px"]
Sep 11 21:53:19.010970 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:53:19.010937    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5dd2da0_d2fc_47b4_8e68_f7d8d2bc3faa.slice/crio-370b379b0b138c6f0383848612a0a3fdd953d3855f679c985421815053a6ac02 WatchSource:0}: Error finding container 370b379b0b138c6f0383848612a0a3fdd953d3855f679c985421815053a6ac02: Status 404 returned error can't find the container with id 370b379b0b138c6f0383848612a0a3fdd953d3855f679c985421815053a6ac02
Sep 11 21:53:19.018053 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.018029    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9swth"]
Sep 11 21:53:19.020846 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:53:19.020825    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07461d1_42d3_43f6_bf6a_89a8663f4ca8.slice/crio-635b9a2414a74e12d950caece7b0508cdc0b8b8878ed5236418444f6c426c810 WatchSource:0}: Error finding container 635b9a2414a74e12d950caece7b0508cdc0b8b8878ed5236418444f6c426c810: Status 404 returned error can't find the container with id 635b9a2414a74e12d950caece7b0508cdc0b8b8878ed5236418444f6c426c810
Sep 11 21:53:19.037417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.037397    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5h829"]
Sep 11 21:53:19.040359 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:53:19.040332    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec958662_21bf_4b7a_89e8_267265fec58f.slice/crio-929c24f5cca2a8a57cf0febe65015cb07de157d93ca64e99e70fda9dec39a99a WatchSource:0}: Error finding container 929c24f5cca2a8a57cf0febe65015cb07de157d93ca64e99e70fda9dec39a99a: Status 404 returned error can't find the container with id 929c24f5cca2a8a57cf0febe65015cb07de157d93ca64e99e70fda9dec39a99a
Sep 11 21:53:19.254232 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.254202    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5h829" event={"ID":"ec958662-21bf-4b7a-89e8-267265fec58f","Type":"ContainerStarted","Data":"641e7d1cea72a35a1f6cce05bbe0686fd41eb1068c59a20385f7a2cea5d53412"}
Sep 11 21:53:19.254425 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.254252    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5h829" event={"ID":"ec958662-21bf-4b7a-89e8-267265fec58f","Type":"ContainerStarted","Data":"929c24f5cca2a8a57cf0febe65015cb07de157d93ca64e99e70fda9dec39a99a"}
Sep 11 21:53:19.254425 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.254274    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:53:19.255432 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.255402    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-628px" event={"ID":"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa","Type":"ContainerStarted","Data":"370b379b0b138c6f0383848612a0a3fdd953d3855f679c985421815053a6ac02"}
Sep 11 21:53:19.256529 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.256510    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9swth" event={"ID":"f07461d1-42d3-43f6-bf6a-89a8663f4ca8","Type":"ContainerStarted","Data":"635b9a2414a74e12d950caece7b0508cdc0b8b8878ed5236418444f6c426c810"}
Sep 11 21:53:19.270069 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.270032    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5h829" podStartSLOduration=65.270019184 podStartE2EDuration="1m5.270019184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:19.269330327 +0000 UTC m=+66.004441671" watchObservedRunningTime="2026-09-11 21:53:19.270019184 +0000 UTC m=+66.005130530"
Sep 11 21:53:19.567071 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.567032    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:53:19.567071 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.567082    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:53:19.567323 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.567130    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:53:19.567323 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.567166    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:53:19.567323 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.567194    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2qrmq\" (UID: \"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:53:19.567323 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.567241    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:53:19.567935 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.567902    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c667eac2-77fb-4708-a43f-ae76bcac6bc1-service-ca-bundle\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:53:19.570594 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.570564    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51337e48-d86f-4001-a58a-c6a2297fe84a-metrics-tls\") pod \"dns-default-qlxmr\" (UID: \"51337e48-d86f-4001-a58a-c6a2297fe84a\") " pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:53:19.570707 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.570636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a452c74-ccd8-4cbf-8850-d9f0bf78ac50-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2qrmq\" (UID: \"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:53:19.571630 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.571607    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c667eac2-77fb-4708-a43f-ae76bcac6bc1-metrics-certs\") pod \"router-default-889c88f7b-28lxw\" (UID: \"c667eac2-77fb-4708-a43f-ae76bcac6bc1\") " pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:53:19.571826 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.571808    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2eecf508-6314-4920-a2c6-4d90d43db253-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-h65fx\" (UID: \"2eecf508-6314-4920-a2c6-4d90d43db253\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:53:19.585760 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.585734    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls\") pod \"image-registry-56d9f5f6cd-crns2\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") " pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:53:19.714525 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.714494    2574 scope.go:117] "RemoveContainer" containerID="2888bf83cff8de6267b86a78f6c73c058dadfedc9228fdd34e76e80b0ff38a5c"
Sep 11 21:53:19.766770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.766743    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qtd28\""
Sep 11 21:53:19.774352 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.774320    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"
Sep 11 21:53:19.782138 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.782118    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m5hpf\""
Sep 11 21:53:19.790406 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.790378    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:53:19.802545 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.802520    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-m59xn\""
Sep 11 21:53:19.811083 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.811061    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:53:19.830215 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.828637    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h6hkz\""
Sep 11 21:53:19.836035 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.835986    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"
Sep 11 21:53:19.845899 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.845879    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8zqrn\""
Sep 11 21:53:19.854213 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:19.854138    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:53:20.082814 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:20.082683    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx"]
Sep 11 21:53:20.173910 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:53:20.173874    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eecf508_6314_4920_a2c6_4d90d43db253.slice/crio-0b1ab7163a93098df775d83f99c625e07ac1cc909c593ba1b7d36990f54b094a WatchSource:0}: Error finding container 0b1ab7163a93098df775d83f99c625e07ac1cc909c593ba1b7d36990f54b094a: Status 404 returned error can't find the container with id 0b1ab7163a93098df775d83f99c625e07ac1cc909c593ba1b7d36990f54b094a
Sep 11 21:53:20.274420 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:20.274264    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx" event={"ID":"2eecf508-6314-4920-a2c6-4d90d43db253","Type":"ContainerStarted","Data":"0b1ab7163a93098df775d83f99c625e07ac1cc909c593ba1b7d36990f54b094a"}
Sep 11 21:53:20.278588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:20.277785    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 21:53:20.278588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:20.277881    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn" event={"ID":"6d54fe9c-b07b-4b7f-96ae-b0af3e131114","Type":"ContainerStarted","Data":"988fda02296e55df5c0994f27598c72e29790b5d221ff2c1219cb535b196c604"}
Sep 11 21:53:20.278588 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:20.278384    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:53:20.296490 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:20.296443    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn" podStartSLOduration=43.655567356 podStartE2EDuration="51.296424246s" totalImagesPullingTime="7.64085689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:20.294514987 +0000 UTC m=+67.029626335" watchObservedRunningTime="2026-09-11 21:53:20.296424246 +0000 UTC m=+67.031535595"
Sep 11 21:53:20.477465 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:20.477429    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq"]
Sep 11 21:53:20.481560 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:20.481534    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-889c88f7b-28lxw"]
Sep 11 21:53:20.482974 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:20.482947    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56d9f5f6cd-crns2"]
Sep 11 21:53:20.484163 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:20.484140    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qlxmr"]
Sep 11 21:53:20.626712 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:20.626689    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-8g8rn"
Sep 11 21:53:20.817844 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:20.817806    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:20.917436 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:20.917387    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:21.016995 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.016893    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:21.164687 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.164625    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:21.287573 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.287380    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qlxmr" event={"ID":"51337e48-d86f-4001-a58a-c6a2297fe84a","Type":"ContainerStarted","Data":"3468337282b0414b8ba8ffc2a39e1b7d8817c776ad050619cbebe1407cdfa4aa"}
Sep 11 21:53:21.291980 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.291923    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2" event={"ID":"c4d96ddd-a046-422c-928b-564b04193b6a","Type":"ContainerStarted","Data":"db6a7edc4016bcbd54bd5c30ec9b4578d80fb7c9bf6416c76dfdd04a0815fab9"}
Sep 11 21:53:21.292629 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.292219    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2" event={"ID":"c4d96ddd-a046-422c-928b-564b04193b6a","Type":"ContainerStarted","Data":"6ae1214b151e1bb539257d3af7095ecf2a99d6f6f0c1dee1c37b810cd0dc9d42"}
Sep 11 21:53:21.293583 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.293540    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:53:21.296247 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.296041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-889c88f7b-28lxw" event={"ID":"c667eac2-77fb-4708-a43f-ae76bcac6bc1","Type":"ContainerStarted","Data":"d8cb7f2e312034441aae75a591d3dc979065893dc19e1f2642753e4c087eeb2c"}
Sep 11 21:53:21.296247 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.296079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-889c88f7b-28lxw" event={"ID":"c667eac2-77fb-4708-a43f-ae76bcac6bc1","Type":"ContainerStarted","Data":"33308682a23acd382e61fa0a7252d1e85c0b479a026531ad5d8a6bd1dad70049"}
Sep 11 21:53:21.299876 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.299359    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9swth" event={"ID":"f07461d1-42d3-43f6-bf6a-89a8663f4ca8","Type":"ContainerStarted","Data":"3bb8fea61c3062f0a1c60f385330dbe808c4fd38b44b25011f773b0a175db6e3"}
Sep 11 21:53:21.299876 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.299405    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9swth" event={"ID":"f07461d1-42d3-43f6-bf6a-89a8663f4ca8","Type":"ContainerStarted","Data":"6a92eae817efbd8f3201b8e4c7102c833e6fafc1f386f0a786aa6eaf18ed2d06"}
Sep 11 21:53:21.301933 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.301897    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq" event={"ID":"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50","Type":"ContainerStarted","Data":"fb68fa8706335ca6939656bf8eb8b0d3ca7d03e36982969b5fc38450deb36787"}
Sep 11 21:53:21.335194 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.334017    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2" podStartSLOduration=67.333997321 podStartE2EDuration="1m7.333997321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:21.333062084 +0000 UTC m=+68.068173430" watchObservedRunningTime="2026-09-11 21:53:21.333997321 +0000 UTC m=+68.069108670"
Sep 11 21:53:21.350409 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.350350    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:21.361342 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.360234    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9swth" podStartSLOduration=65.836944076 podStartE2EDuration="1m7.36021725s" totalImagesPullingTime="1.523273174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:21.360057505 +0000 UTC m=+68.095168878" watchObservedRunningTime="2026-09-11 21:53:21.36021725 +0000 UTC m=+68.095328597"
Sep 11 21:53:21.403650 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.403537    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-889c88f7b-28lxw" podStartSLOduration=53.403517675 podStartE2EDuration="53.403517675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:21.403020366 +0000 UTC m=+68.138131742" watchObservedRunningTime="2026-09-11 21:53:21.403517675 +0000 UTC m=+68.138629023"
Sep 11 21:53:21.482019 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.481968    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:21.555526 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.555444    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:21.655749 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.655550    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:21.812241 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.811994    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:53:21.815040 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:21.814832    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:53:22.306485 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:22.306455    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:53:22.307496 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:22.307474    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-889c88f7b-28lxw"
Sep 11 21:53:22.498239 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:22.498205    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:22.694409 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:22.694362    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:22.956864 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:22.956772    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:23.117048 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:23.117009    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:23.174104 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:23.174070    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:24.112322 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:24.112265    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:24.175139 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:24.175106    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:25.117289 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.117250    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:25.221525 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.221437    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:25.290042 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.290006    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:25.318783 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.318749    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq" event={"ID":"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50","Type":"ContainerStarted","Data":"386170b2d468f50c5a73e4f09c9b207bacf1b0a48c315cfb5676086dcdf809cb"}
Sep 11 21:53:25.318939 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.318811    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq" event={"ID":"1a452c74-ccd8-4cbf-8850-d9f0bf78ac50","Type":"ContainerStarted","Data":"b36e6813f0482c95adf94c2fdc9edd77dba2401c05c93e4f7738089d574ada91"}
Sep 11 21:53:25.320322 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.320291    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-628px" event={"ID":"b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa","Type":"ContainerStarted","Data":"ad874848d5c16105e538c5bf9bae2fae6d4bcb5c4bff722b985d4c477cc58f19"}
Sep 11 21:53:25.322072 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.322051    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qlxmr" event={"ID":"51337e48-d86f-4001-a58a-c6a2297fe84a","Type":"ContainerStarted","Data":"330c4dd3bfe6e173901bb0d96757eaac755b7fd0e2e9796dbb01907b1f9d6db7"}
Sep 11 21:53:25.322172 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.322088    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qlxmr" event={"ID":"51337e48-d86f-4001-a58a-c6a2297fe84a","Type":"ContainerStarted","Data":"ab19d8dc8760b3c590dc455e16e3e4cf953a1a355f337639d7fbefda85866526"}
Sep 11 21:53:25.322172 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.322114    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:53:25.323984 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.323965    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx" event={"ID":"2eecf508-6314-4920-a2c6-4d90d43db253","Type":"ContainerStarted","Data":"48c876a8642717bf20995a46f7f2ccbb8c9218bad4aed65fff7401ded76ec540"}
Sep 11 21:53:25.338501 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.338458    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2qrmq" podStartSLOduration=52.195967202 podStartE2EDuration="56.338445541s" totalImagesPullingTime="4.142478339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:25.3361459 +0000 UTC m=+72.071257247" watchObservedRunningTime="2026-09-11 21:53:25.338445541 +0000 UTC m=+72.073556887"
Sep 11 21:53:25.351898 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.351823    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-h65fx" podStartSLOduration=52.792150105 podStartE2EDuration="57.35180894s" totalImagesPullingTime="4.559658835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:25.351559908 +0000 UTC m=+72.086671260" watchObservedRunningTime="2026-09-11 21:53:25.35180894 +0000 UTC m=+72.086920286"
Sep 11 21:53:25.369259 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.369218    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qlxmr" podStartSLOduration=34.171913373 podStartE2EDuration="38.369202657s" totalImagesPullingTime="4.197289284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:25.368862845 +0000 UTC m=+72.103974190" watchObservedRunningTime="2026-09-11 21:53:25.369202657 +0000 UTC m=+72.104314002"
Sep 11 21:53:25.382831 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.382784    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-628px" podStartSLOduration=65.64751379 podStartE2EDuration="1m11.382766515s" totalImagesPullingTime="5.735252725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:25.382670136 +0000 UTC m=+72.117781481" watchObservedRunningTime="2026-09-11 21:53:25.382766515 +0000 UTC m=+72.117877862"
Sep 11 21:53:25.719528 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:25.719496    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:26.623139 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:26.623097    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:26.724114 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:26.724075    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:26.779114 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:26.779082    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:27.007163 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:27.007077    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:27.068367 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:27.068333    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:29.334498 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.334455    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-88cb58d6-jdbtb"]
Sep 11 21:53:29.366914 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.366884    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88cb58d6-jdbtb"]
Sep 11 21:53:29.367074 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.366977    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.370187 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.370154    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 21:53:29.370341 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.370207    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 21:53:29.370341 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.370213    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 21:53:29.370341 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.370238    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 21:53:29.371389 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.371346    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 21:53:29.371389 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.371379    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q495q\""
Sep 11 21:53:29.371614 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.371364    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 21:53:29.371614 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.371447    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 21:53:29.439809 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.439769    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-service-ca\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.439809 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.439809    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-oauth-serving-cert\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.440069 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.439830    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcj58\" (UniqueName: \"kubernetes.io/projected/b7af72c2-db69-45aa-ba08-b8115ee30163-kube-api-access-gcj58\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.440069 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.439848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7af72c2-db69-45aa-ba08-b8115ee30163-console-serving-cert\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.440069 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.439940    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7af72c2-db69-45aa-ba08-b8115ee30163-console-oauth-config\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.440069 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.439980    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-console-config\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.540886 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.540849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-service-ca\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.540886 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.540887    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-oauth-serving-cert\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.541122 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.540908    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcj58\" (UniqueName: \"kubernetes.io/projected/b7af72c2-db69-45aa-ba08-b8115ee30163-kube-api-access-gcj58\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.541122 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.540932    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7af72c2-db69-45aa-ba08-b8115ee30163-console-serving-cert\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.541122 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.541082    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7af72c2-db69-45aa-ba08-b8115ee30163-console-oauth-config\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.541268 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.541130    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-console-config\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.541655 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.541630    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-oauth-serving-cert\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.541655 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.541648    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-service-ca\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.541811 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.541799    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-console-config\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.543303 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.543278    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7af72c2-db69-45aa-ba08-b8115ee30163-console-serving-cert\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.543461 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.543442    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7af72c2-db69-45aa-ba08-b8115ee30163-console-oauth-config\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.549120 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.549098    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcj58\" (UniqueName: \"kubernetes.io/projected/b7af72c2-db69-45aa-ba08-b8115ee30163-kube-api-access-gcj58\") pod \"console-88cb58d6-jdbtb\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") " pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.676243 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.676211    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:29.699428 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.699384    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:29.764613 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.764569    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:29.977378 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:29.977293    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88cb58d6-jdbtb"]
Sep 11 21:53:29.980252 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:53:29.980227    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7af72c2_db69_45aa_ba08_b8115ee30163.slice/crio-9af45b9ffbbab55066d4ca2d3891ec5542d63a3ccb1d6f07667b6ec3e5ac4389 WatchSource:0}: Error finding container 9af45b9ffbbab55066d4ca2d3891ec5542d63a3ccb1d6f07667b6ec3e5ac4389: Status 404 returned error can't find the container with id 9af45b9ffbbab55066d4ca2d3891ec5542d63a3ccb1d6f07667b6ec3e5ac4389
Sep 11 21:53:30.345912 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:30.345884    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88cb58d6-jdbtb" event={"ID":"b7af72c2-db69-45aa-ba08-b8115ee30163","Type":"ContainerStarted","Data":"9af45b9ffbbab55066d4ca2d3891ec5542d63a3ccb1d6f07667b6ec3e5ac4389"}
Sep 11 21:53:31.497299 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:31.497259    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:31.598809 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:31.598773    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:32.323990 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.323952    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:32.358872 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.358838    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"]
Sep 11 21:53:32.382693 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.382661    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"]
Sep 11 21:53:32.382842 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.382781    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:32.387032 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.387006    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 21:53:32.387139 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.387061    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 21:53:32.387139 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.387075    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 21:53:32.387239 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.387015    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ckcpd\""
Sep 11 21:53:32.406426 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.406393    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:32.464468 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.464061    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab511a00-2391-40c9-a790-663015e8d534-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-qt8pf\" (UID: \"ab511a00-2391-40c9-a790-663015e8d534\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:32.464468 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.464104    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxlk\" (UniqueName: \"kubernetes.io/projected/ab511a00-2391-40c9-a790-663015e8d534-kube-api-access-rxxlk\") pod \"prometheus-operator-68bcd7f987-qt8pf\" (UID: \"ab511a00-2391-40c9-a790-663015e8d534\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:32.464468 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.464154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab511a00-2391-40c9-a790-663015e8d534-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-qt8pf\" (UID: \"ab511a00-2391-40c9-a790-663015e8d534\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:32.464468 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.464208    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab511a00-2391-40c9-a790-663015e8d534-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-qt8pf\" (UID: \"ab511a00-2391-40c9-a790-663015e8d534\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:32.523773 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.523737    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:32.564791 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.564759    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab511a00-2391-40c9-a790-663015e8d534-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-qt8pf\" (UID: \"ab511a00-2391-40c9-a790-663015e8d534\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:32.564960 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.564827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab511a00-2391-40c9-a790-663015e8d534-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-qt8pf\" (UID: \"ab511a00-2391-40c9-a790-663015e8d534\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:32.564960 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.564854    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxlk\" (UniqueName: \"kubernetes.io/projected/ab511a00-2391-40c9-a790-663015e8d534-kube-api-access-rxxlk\") pod \"prometheus-operator-68bcd7f987-qt8pf\" (UID: \"ab511a00-2391-40c9-a790-663015e8d534\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:32.564960 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.564893    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab511a00-2391-40c9-a790-663015e8d534-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-qt8pf\" (UID: \"ab511a00-2391-40c9-a790-663015e8d534\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:32.564960 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:32.564925    2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 11 21:53:32.565118 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:32.565010    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab511a00-2391-40c9-a790-663015e8d534-prometheus-operator-tls podName:ab511a00-2391-40c9-a790-663015e8d534 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:33.064990356 +0000 UTC m=+79.800101679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ab511a00-2391-40c9-a790-663015e8d534-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-qt8pf" (UID: "ab511a00-2391-40c9-a790-663015e8d534") : secret "prometheus-operator-tls" not found
Sep 11 21:53:32.565575 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.565550    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab511a00-2391-40c9-a790-663015e8d534-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-qt8pf\" (UID: \"ab511a00-2391-40c9-a790-663015e8d534\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:32.567515 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.567490    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab511a00-2391-40c9-a790-663015e8d534-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-qt8pf\" (UID: \"ab511a00-2391-40c9-a790-663015e8d534\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:32.574331 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.574257    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxlk\" (UniqueName: \"kubernetes.io/projected/ab511a00-2391-40c9-a790-663015e8d534-kube-api-access-rxxlk\") pod \"prometheus-operator-68bcd7f987-qt8pf\" (UID: \"ab511a00-2391-40c9-a790-663015e8d534\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:32.625924 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.625892    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:32.726828 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:32.726787    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:33.068785 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:33.068752    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab511a00-2391-40c9-a790-663015e8d534-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-qt8pf\" (UID: \"ab511a00-2391-40c9-a790-663015e8d534\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:33.071028 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:33.070998    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab511a00-2391-40c9-a790-663015e8d534-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-qt8pf\" (UID: \"ab511a00-2391-40c9-a790-663015e8d534\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:33.294691 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:33.294618    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"
Sep 11 21:53:33.360384 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:33.360350    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88cb58d6-jdbtb" event={"ID":"b7af72c2-db69-45aa-ba08-b8115ee30163","Type":"ContainerStarted","Data":"f749669cb4fc10e1c394b3fc2bdb4a3d52b41b86b25f83a281836aeaa61deebf"}
Sep 11 21:53:33.380872 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:33.380830    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-88cb58d6-jdbtb" podStartSLOduration=1.464231616 podStartE2EDuration="4.380813687s" totalImagesPullingTime="2.916582071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:33.379802781 +0000 UTC m=+80.114914128" watchObservedRunningTime="2026-09-11 21:53:33.380813687 +0000 UTC m=+80.115925034"
Sep 11 21:53:33.562606 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:33.562544    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf"]
Sep 11 21:53:33.565826 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:53:33.565803    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab511a00_2391_40c9_a790_663015e8d534.slice/crio-ebc2bb9785042d9466e08b2ab21b3a22b0d029976579d2277440ec349e5a173b WatchSource:0}: Error finding container ebc2bb9785042d9466e08b2ab21b3a22b0d029976579d2277440ec349e5a173b: Status 404 returned error can't find the container with id ebc2bb9785042d9466e08b2ab21b3a22b0d029976579d2277440ec349e5a173b
Sep 11 21:53:33.712361 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:33.712329    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:33.772107 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:33.772082    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:34.366201 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:34.366158    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf" event={"ID":"ab511a00-2391-40c9-a790-663015e8d534","Type":"ContainerStarted","Data":"ebc2bb9785042d9466e08b2ab21b3a22b0d029976579d2277440ec349e5a173b"}
Sep 11 21:53:35.092413 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:35.092377    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:35.330329 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:35.330237    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qlxmr"
Sep 11 21:53:35.372810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:35.372779    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf" event={"ID":"ab511a00-2391-40c9-a790-663015e8d534","Type":"ContainerStarted","Data":"bac55846b9f1c0f515956d06c4432c09462c6007dab9ac613821f5e3bc79c311"}
Sep 11 21:53:35.372955 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:35.372831    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf" event={"ID":"ab511a00-2391-40c9-a790-663015e8d534","Type":"ContainerStarted","Data":"ab23d858490f2530d1cdf7decc884723d119ae47a5da16c8fcc21987c1d2d0a7"}
Sep 11 21:53:35.389432 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:35.389383    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qt8pf" podStartSLOduration=2.106075129 podStartE2EDuration="3.389369085s" totalImagesPullingTime="1.283293956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:35.388612067 +0000 UTC m=+82.123723440" watchObservedRunningTime="2026-09-11 21:53:35.389369085 +0000 UTC m=+82.124480431"
Sep 11 21:53:36.709582 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:36.709457    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:36.767264 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:36.767235    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:37.463961 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.463932    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:37.684101 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.684072    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:37.708841 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.708807    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-br9dj"]
Sep 11 21:53:37.711804 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.711783    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.714631 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.714561    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b2br2\""
Sep 11 21:53:37.714631 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.714588    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 21:53:37.714631 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.714591    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 21:53:37.714827 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.714591    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 21:53:37.771746 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.771708    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:37.803790 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.803767    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7571fe76-5cf1-447d-a61d-9d49e14657ea-sys\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.803906 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.803802    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-textfile\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.803906 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.803821    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b29f5\" (UniqueName: \"kubernetes.io/projected/7571fe76-5cf1-447d-a61d-9d49e14657ea-kube-api-access-b29f5\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.804003 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.803895    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.804003 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.803974    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-wtmp\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.804003 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.803990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7571fe76-5cf1-447d-a61d-9d49e14657ea-root\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.804157 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.804006    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-tls\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.804157 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.804024    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7571fe76-5cf1-447d-a61d-9d49e14657ea-metrics-client-ca\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.804157 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.804073    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.827278 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.827254    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:37.905183 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905163    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-tls\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.905290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7571fe76-5cf1-447d-a61d-9d49e14657ea-metrics-client-ca\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.905290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905223    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.905290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905269    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7571fe76-5cf1-447d-a61d-9d49e14657ea-sys\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.905452 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905290    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-textfile\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.905452 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905329    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b29f5\" (UniqueName: \"kubernetes.io/projected/7571fe76-5cf1-447d-a61d-9d49e14657ea-kube-api-access-b29f5\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.905452 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:37.905351    2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 21:53:37.905452 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905376    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.905452 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905375    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7571fe76-5cf1-447d-a61d-9d49e14657ea-sys\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.905452 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-wtmp\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.905726 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7571fe76-5cf1-447d-a61d-9d49e14657ea-root\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.905726 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905529    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-wtmp\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.905726 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905533    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7571fe76-5cf1-447d-a61d-9d49e14657ea-root\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.905726 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:37.905572    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-tls podName:7571fe76-5cf1-447d-a61d-9d49e14657ea nodeName:}" failed. No retries permitted until 2026-09-11 21:53:38.40555154 +0000 UTC m=+85.140663070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-tls") pod "node-exporter-br9dj" (UID: "7571fe76-5cf1-447d-a61d-9d49e14657ea") : secret "node-exporter-tls" not found
Sep 11 21:53:37.905854 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905750    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-textfile\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.905928 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.905905    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.906126 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.906107    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7571fe76-5cf1-447d-a61d-9d49e14657ea-metrics-client-ca\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.907696 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.907675    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.917368 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.917348    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b29f5\" (UniqueName: \"kubernetes.io/projected/7571fe76-5cf1-447d-a61d-9d49e14657ea-kube-api-access-b29f5\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:37.934528 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:37.934504    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:38.066882 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:38.066800    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:38.162997 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:38.162969    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:38.263811 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:38.263779    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:38.364089 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:38.364047    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:38.410245 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:38.410221    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-tls\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:38.412486 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:38.412469    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7571fe76-5cf1-447d-a61d-9d49e14657ea-node-exporter-tls\") pod \"node-exporter-br9dj\" (UID: \"7571fe76-5cf1-447d-a61d-9d49e14657ea\") " pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:38.621805 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:38.621747    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-br9dj"
Sep 11 21:53:38.631704 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:53:38.631682    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7571fe76_5cf1_447d_a61d_9d49e14657ea.slice/crio-78f1707e2f484f3332790eae297987484040dbc10fd8c704bdbf2614003ad5c4 WatchSource:0}: Error finding container 78f1707e2f484f3332790eae297987484040dbc10fd8c704bdbf2614003ad5c4: Status 404 returned error can't find the container with id 78f1707e2f484f3332790eae297987484040dbc10fd8c704bdbf2614003ad5c4
Sep 11 21:53:38.709892 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:38.709858    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:38.799170 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:38.799136    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:38.913637 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:38.913558    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:39.121959 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.121924    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:39.390183 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.390148    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-br9dj" event={"ID":"7571fe76-5cf1-447d-a61d-9d49e14657ea","Type":"ContainerStarted","Data":"78f1707e2f484f3332790eae297987484040dbc10fd8c704bdbf2614003ad5c4"}
Sep 11 21:53:39.677237 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.677164    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:39.677237 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.677226    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:39.682657 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.682640    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:39.724065 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.724039    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"]
Sep 11 21:53:39.726811 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.726790    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.729660 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.729635    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 21:53:39.729785 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.729702    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5b2w2\""
Sep 11 21:53:39.730117 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.730093    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 21:53:39.730240 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.730121    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 21:53:39.730240 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.730171    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-63tvbt14p7i9k\""
Sep 11 21:53:39.730240 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.730222    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 21:53:39.730458 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.730358    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 21:53:39.735968 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.735943    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:39.741131 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.741111    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"]
Sep 11 21:53:39.821174 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.821153    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hcdf\" (UniqueName: \"kubernetes.io/projected/d094fef9-0e11-400b-aca0-ff688ec8ab79-kube-api-access-8hcdf\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.821527 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.821192    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.821527 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.821212    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d094fef9-0e11-400b-aca0-ff688ec8ab79-metrics-client-ca\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.821527 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.821261    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-tls\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.821527 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.821298    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.821527 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.821363    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.821527 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.821397    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.821527 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.821482    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-grpc-tls\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.922590 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.922534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.922590 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.922565    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d094fef9-0e11-400b-aca0-ff688ec8ab79-metrics-client-ca\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.922590 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.922586    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-tls\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.922804 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.922607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.922804 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.922671    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.922804 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.922706    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.922804 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.922747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-grpc-tls\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.923010 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.922888    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hcdf\" (UniqueName: \"kubernetes.io/projected/d094fef9-0e11-400b-aca0-ff688ec8ab79-kube-api-access-8hcdf\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.923355 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.923335    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d094fef9-0e11-400b-aca0-ff688ec8ab79-metrics-client-ca\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.925091 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.925040    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.925472 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.925442    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.925573 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.925504    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.925634 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.925577    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.925688 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.925653    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-thanos-querier-tls\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.925688 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.925673    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d094fef9-0e11-400b-aca0-ff688ec8ab79-secret-grpc-tls\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:39.931789 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:39.931768    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hcdf\" (UniqueName: \"kubernetes.io/projected/d094fef9-0e11-400b-aca0-ff688ec8ab79-kube-api-access-8hcdf\") pod \"thanos-querier-5fb545d8cf-xz62b\" (UID: \"d094fef9-0e11-400b-aca0-ff688ec8ab79\") " pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:40.100940 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:40.100910    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:40.374930 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:40.374908    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"]
Sep 11 21:53:40.375917 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:53:40.375891    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd094fef9_0e11_400b_aca0_ff688ec8ab79.slice/crio-06a4a9be09398896b39ae1d5dec8b756bdda41aeee1e527ced2bd21bc6495a5c WatchSource:0}: Error finding container 06a4a9be09398896b39ae1d5dec8b756bdda41aeee1e527ced2bd21bc6495a5c: Status 404 returned error can't find the container with id 06a4a9be09398896b39ae1d5dec8b756bdda41aeee1e527ced2bd21bc6495a5c
Sep 11 21:53:40.395477 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:40.395454    2574 generic.go:356] "Generic (PLEG): container finished" podID="7571fe76-5cf1-447d-a61d-9d49e14657ea" containerID="93731dda5ef8c2baa87646b19a776a8a74f873c5780e320fae26be96c80b741d" exitCode=0
Sep 11 21:53:40.395572 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:40.395525    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-br9dj" event={"ID":"7571fe76-5cf1-447d-a61d-9d49e14657ea","Type":"ContainerDied","Data":"93731dda5ef8c2baa87646b19a776a8a74f873c5780e320fae26be96c80b741d"}
Sep 11 21:53:40.396789 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:40.396765    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b" event={"ID":"d094fef9-0e11-400b-aca0-ff688ec8ab79","Type":"ContainerStarted","Data":"06a4a9be09398896b39ae1d5dec8b756bdda41aeee1e527ced2bd21bc6495a5c"}
Sep 11 21:53:40.400945 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:40.400926    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:53:40.634473 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:40.634397    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:40.735379 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:40.735348    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:40.857377 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:40.857342    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:41.191529 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:41.191485    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:41.404277 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:41.404238    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-br9dj" event={"ID":"7571fe76-5cf1-447d-a61d-9d49e14657ea","Type":"ContainerStarted","Data":"2d25f49a5260132d5cf6be646181c24b0e16a0052800380a07ade21103dd6f1b"}
Sep 11 21:53:41.404476 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:41.404303    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-br9dj" event={"ID":"7571fe76-5cf1-447d-a61d-9d49e14657ea","Type":"ContainerStarted","Data":"324ac765167a3f22ab96d2649f1f14ffe69a2de01b491a2015357d9bdbd81dec"}
Sep 11 21:53:41.430408 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:41.430346    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-br9dj" podStartSLOduration=3.4016660229999998 podStartE2EDuration="4.430324711s" totalImagesPullingTime="961.177591ms" totalInitContainerRuntime="67.481097ms" isStatefulPod=true podCreationTimestamp="2026-09-11 21:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:41.427639473 +0000 UTC m=+88.162750821" watchObservedRunningTime="2026-09-11 21:53:41.430324711 +0000 UTC m=+88.165436054"
Sep 11 21:53:41.512212 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:41.512127    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:41.569668 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:41.569611    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:42.410903 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.410872    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b" event={"ID":"d094fef9-0e11-400b-aca0-ff688ec8ab79","Type":"ContainerStarted","Data":"4c0fe4ad3ddd212740595aebf88ed4cf9908e2b5d62b433339d1e7e12041da94"}
Sep 11 21:53:42.411288 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.410938    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b" event={"ID":"d094fef9-0e11-400b-aca0-ff688ec8ab79","Type":"ContainerStarted","Data":"2a6b1db2e8651d3e6d96fcb1749c054ee3a0b145a3af624273f32c7dc913770b"}
Sep 11 21:53:42.411288 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.410964    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b" event={"ID":"d094fef9-0e11-400b-aca0-ff688ec8ab79","Type":"ContainerStarted","Data":"127c519012fbef225710aca7ce642bf2f2c6f80d2dbab8336cda7ec1d2ed2098"}
Sep 11 21:53:42.497131 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.497095    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"]
Sep 11 21:53:42.499480 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.499458    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"
Sep 11 21:53:42.501494 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.501466    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:42.502240 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.502219    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zgbzb\""
Sep 11 21:53:42.502360 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.502222    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 21:53:42.510857 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.510837    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"]
Sep 11 21:53:42.556294 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.556270    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c4bcd6df5-t8fsw"]
Sep 11 21:53:42.558482 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.558464    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.566960 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.566938    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 21:53:42.569173 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.569151    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4bcd6df5-t8fsw"]
Sep 11 21:53:42.572832 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.572811    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:42.643981 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.643959    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-config\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.644104 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.643995    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2466e0c1-64f6-4591-b3d2-cf3c50ece04c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-m582g\" (UID: \"2466e0c1-64f6-4591-b3d2-cf3c50ece04c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"
Sep 11 21:53:42.644104 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.644021    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-trusted-ca-bundle\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.644216 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.644116    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-oauth-serving-cert\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.644266 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.644182    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g85jl\" (UniqueName: \"kubernetes.io/projected/2466e0c1-64f6-4591-b3d2-cf3c50ece04c-kube-api-access-g85jl\") pod \"monitoring-plugin-5d85778b9d-m582g\" (UID: \"2466e0c1-64f6-4591-b3d2-cf3c50ece04c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"
Sep 11 21:53:42.644330 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.644267    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-oauth-config\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.644387 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.644333    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-serving-cert\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.644387 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.644367    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z97vd\" (UniqueName: \"kubernetes.io/projected/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-kube-api-access-z97vd\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.644387 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.644384    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-service-ca\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.744999 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.744928    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-service-ca\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.744999 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.744977    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-config\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.745190 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.745018    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2466e0c1-64f6-4591-b3d2-cf3c50ece04c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-m582g\" (UID: \"2466e0c1-64f6-4591-b3d2-cf3c50ece04c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"
Sep 11 21:53:42.745190 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.745044    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-trusted-ca-bundle\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.745190 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.745098    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-oauth-serving-cert\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.745190 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.745140    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g85jl\" (UniqueName: \"kubernetes.io/projected/2466e0c1-64f6-4591-b3d2-cf3c50ece04c-kube-api-access-g85jl\") pod \"monitoring-plugin-5d85778b9d-m582g\" (UID: \"2466e0c1-64f6-4591-b3d2-cf3c50ece04c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"
Sep 11 21:53:42.745190 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.745172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-oauth-config\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.745461 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:42.745188    2574 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 11 21:53:42.745461 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.745214    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-serving-cert\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.745461 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:53:42.745257    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2466e0c1-64f6-4591-b3d2-cf3c50ece04c-monitoring-plugin-cert podName:2466e0c1-64f6-4591-b3d2-cf3c50ece04c nodeName:}" failed. No retries permitted until 2026-09-11 21:53:43.245234942 +0000 UTC m=+89.980346277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/2466e0c1-64f6-4591-b3d2-cf3c50ece04c-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-m582g" (UID: "2466e0c1-64f6-4591-b3d2-cf3c50ece04c") : secret "monitoring-plugin-cert" not found
Sep 11 21:53:42.745461 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.745300    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z97vd\" (UniqueName: \"kubernetes.io/projected/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-kube-api-access-z97vd\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.745721 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.745689    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-service-ca\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.745877 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.745853    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-config\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.746070 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.746045    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-trusted-ca-bundle\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.746209 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.746191    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-oauth-serving-cert\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.747830 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.747802    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-oauth-config\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.747972 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.747954    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-serving-cert\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.754756 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.754729    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g85jl\" (UniqueName: \"kubernetes.io/projected/2466e0c1-64f6-4591-b3d2-cf3c50ece04c-kube-api-access-g85jl\") pod \"monitoring-plugin-5d85778b9d-m582g\" (UID: \"2466e0c1-64f6-4591-b3d2-cf3c50ece04c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"
Sep 11 21:53:42.755034 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.755014    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z97vd\" (UniqueName: \"kubernetes.io/projected/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-kube-api-access-z97vd\") pod \"console-7c4bcd6df5-t8fsw\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") " pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.869056 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.869027    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:42.909374 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:42.909296    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:43.150160 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.150136    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4bcd6df5-t8fsw"]
Sep 11 21:53:43.152622 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:53:43.152594    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7000c6aa_6a4f_4922_981b_8a5ed4b51a08.slice/crio-e631cdb533631bab7cff5a0544dd0fe47659e4447aa660e0b1206bf6795d3912 WatchSource:0}: Error finding container e631cdb533631bab7cff5a0544dd0fe47659e4447aa660e0b1206bf6795d3912: Status 404 returned error can't find the container with id e631cdb533631bab7cff5a0544dd0fe47659e4447aa660e0b1206bf6795d3912
Sep 11 21:53:43.249077 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.249016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2466e0c1-64f6-4591-b3d2-cf3c50ece04c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-m582g\" (UID: \"2466e0c1-64f6-4591-b3d2-cf3c50ece04c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"
Sep 11 21:53:43.251189 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.251163    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2466e0c1-64f6-4591-b3d2-cf3c50ece04c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-m582g\" (UID: \"2466e0c1-64f6-4591-b3d2-cf3c50ece04c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"
Sep 11 21:53:43.313966 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.313940    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:53:43.408585 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.408551    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"
Sep 11 21:53:43.420268 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.420236    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4bcd6df5-t8fsw" event={"ID":"7000c6aa-6a4f-4922-981b-8a5ed4b51a08","Type":"ContainerStarted","Data":"5742b4d3c88714e2dd8e906f62538a9c02350d311763b8cca30321f2fc59dbcc"}
Sep 11 21:53:43.420715 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.420300    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4bcd6df5-t8fsw" event={"ID":"7000c6aa-6a4f-4922-981b-8a5ed4b51a08","Type":"ContainerStarted","Data":"e631cdb533631bab7cff5a0544dd0fe47659e4447aa660e0b1206bf6795d3912"}
Sep 11 21:53:43.423291 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.423264    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b" event={"ID":"d094fef9-0e11-400b-aca0-ff688ec8ab79","Type":"ContainerStarted","Data":"dae627a43574701264f990507d51e92bbcfa65ba56b89adf43c0f7814a005213"}
Sep 11 21:53:43.423441 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.423339    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b" event={"ID":"d094fef9-0e11-400b-aca0-ff688ec8ab79","Type":"ContainerStarted","Data":"bf742a4c710ab96a3f4eb3fbe6d44ad98ce16f842d233608be77695de8641425"}
Sep 11 21:53:43.423441 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.423366    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b" event={"ID":"d094fef9-0e11-400b-aca0-ff688ec8ab79","Type":"ContainerStarted","Data":"480a81294610359d49712630fdd5121ad2bac6966cc55bbbd3c4cb89f2b59448"}
Sep 11 21:53:43.423544 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.423480    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:43.442551 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.442507    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c4bcd6df5-t8fsw" podStartSLOduration=1.442492074 podStartE2EDuration="1.442492074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:43.441429593 +0000 UTC m=+90.176540940" watchObservedRunningTime="2026-09-11 21:53:43.442492074 +0000 UTC m=+90.177603462"
Sep 11 21:53:43.463080 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.463022    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b" podStartSLOduration=2.137249385 podStartE2EDuration="4.463002332s" totalImagesPullingTime="2.325752947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:43.462122802 +0000 UTC m=+90.197234149" watchObservedRunningTime="2026-09-11 21:53:43.463002332 +0000 UTC m=+90.198113678"
Sep 11 21:53:43.587335 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.587292    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:43.670463 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.670434    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56d9f5f6cd-crns2"]
Sep 11 21:53:43.684396 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.683631    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"]
Sep 11 21:53:43.686087 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.685831    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:43.686291 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:53:43.686253    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2466e0c1_64f6_4591_b3d2_cf3c50ece04c.slice/crio-068c3a9a104af7a04a6d30b00c7796b4fbdb93d9ff3c681baad25efee5bfa736 WatchSource:0}: Error finding container 068c3a9a104af7a04a6d30b00c7796b4fbdb93d9ff3c681baad25efee5bfa736: Status 404 returned error can't find the container with id 068c3a9a104af7a04a6d30b00c7796b4fbdb93d9ff3c681baad25efee5bfa736
Sep 11 21:53:43.765843 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.765783    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:43.853525 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.853442    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:43.958794 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.958760    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 21:53:43.960177 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.960141    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:43.962805 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.962786    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:43.966521 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.966500    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 21:53:43.966840 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.966810    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 21:53:43.967030 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.967009    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 21:53:43.967136 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.967074    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 21:53:43.967748 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.967727    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jzhhc\""
Sep 11 21:53:43.967828 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.967749    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 21:53:43.967959 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.967943    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 21:53:43.968060 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.968042    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 21:53:43.968125 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.968086    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 21:53:43.968185 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.968160    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 21:53:43.968395 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.968374    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 21:53:43.968487 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.968053    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d4lfk5i11iqhs\""
Sep 11 21:53:43.968782 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.968750    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 21:53:43.973882 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.973860    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 21:53:43.974117 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.974093    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 21:53:43.974633 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.974609    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 21:53:43.981656 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.981245    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 21:53:43.989451 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:43.989429    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 21:53:44.054885 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.054841    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:44.056158 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056131    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056291 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056170    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056291 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056195    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056291 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056225    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056291 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/222facd6-c176-4d8b-964b-6c54974f04fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056472 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056347    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056472 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056412    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056472 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056585 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056585 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056585 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056547    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056688 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056581    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056688 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056608    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056688 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-config\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056688 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056670    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056818 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056818 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056783    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbztx\" (UniqueName: \"kubernetes.io/projected/222facd6-c176-4d8b-964b-6c54974f04fc-kube-api-access-pbztx\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056883 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056832    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/222facd6-c176-4d8b-964b-6c54974f04fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056883 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.056883 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.056868    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.157862 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.157737    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.157862 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.157835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.157862 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.157863    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158112 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.157895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158112 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.157921    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158112 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.157947    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158112 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.157967    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158112 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.157999    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-config\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158112 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.158024    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158112 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.158063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158112 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.158091    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbztx\" (UniqueName: \"kubernetes.io/projected/222facd6-c176-4d8b-964b-6c54974f04fc-kube-api-access-pbztx\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158518 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.158143    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/222facd6-c176-4d8b-964b-6c54974f04fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158518 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.158167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158518 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.158192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158518 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.158230    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158518 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.158255    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158518 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.158283    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158518 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.158326    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158518 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.158358    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/222facd6-c176-4d8b-964b-6c54974f04fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158518 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.158387    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.158932 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.158858    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.159029 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.159006    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.159178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.159149    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.159447 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.159420    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.159545 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.159464    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.159827 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.159797    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.161151 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.160850    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.161981 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.161933    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.163333 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.163207    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.163333 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.163207    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.164374 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.163700    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.164374 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.164120    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/222facd6-c176-4d8b-964b-6c54974f04fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.164374 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.164131    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-config\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.164721 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.164462    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.164721 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.164572    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.164826 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.164804    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.165135 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.165105    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/222facd6-c176-4d8b-964b-6c54974f04fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.165238 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.165218    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.165956 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.165934    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.175822 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.175780    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbztx\" (UniqueName: \"kubernetes.io/projected/222facd6-c176-4d8b-964b-6c54974f04fc-kube-api-access-pbztx\") pod \"prometheus-k8s-0\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.285835 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.285796    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:44.432935 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.432830    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g" event={"ID":"2466e0c1-64f6-4591-b3d2-cf3c50ece04c","Type":"ContainerStarted","Data":"068c3a9a104af7a04a6d30b00c7796b4fbdb93d9ff3c681baad25efee5bfa736"}
Sep 11 21:53:44.588130 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:44.588100    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 21:53:44.591348 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:53:44.591294    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222facd6_c176_4d8b_964b_6c54974f04fc.slice/crio-f9bf4783601393bf64ce92a81a38b5d969b37e68198186fbe61de52bf07f3a74 WatchSource:0}: Error finding container f9bf4783601393bf64ce92a81a38b5d969b37e68198186fbe61de52bf07f3a74: Status 404 returned error can't find the container with id f9bf4783601393bf64ce92a81a38b5d969b37e68198186fbe61de52bf07f3a74
Sep 11 21:53:45.439038 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:45.438999    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g" event={"ID":"2466e0c1-64f6-4591-b3d2-cf3c50ece04c","Type":"ContainerStarted","Data":"5486305e613a12e45cdb5764d355f68b86f493874617da72a63bd23deebe71d3"}
Sep 11 21:53:45.439535 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:45.439069    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"
Sep 11 21:53:45.440523 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:45.440497    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerStarted","Data":"f9bf4783601393bf64ce92a81a38b5d969b37e68198186fbe61de52bf07f3a74"}
Sep 11 21:53:45.444500 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:45.444466    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g"
Sep 11 21:53:45.455080 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:45.454980    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-m582g" podStartSLOduration=2.082431298 podStartE2EDuration="3.454961662s" totalImagesPullingTime="1.372530364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:45.454261693 +0000 UTC m=+92.189373040" watchObservedRunningTime="2026-09-11 21:53:45.454961662 +0000 UTC m=+92.190073009"
Sep 11 21:53:46.226301 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:46.226229    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:46.327368 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:46.327337    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:46.392267 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:46.392235    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:46.446674 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:46.446646    2574 generic.go:356] "Generic (PLEG): container finished" podID="222facd6-c176-4d8b-964b-6c54974f04fc" containerID="20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44" exitCode=0
Sep 11 21:53:46.447084 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:46.446684    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerDied","Data":"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"}
Sep 11 21:53:48.169032 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:48.168985    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:48.218327 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:48.218253    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:48.719693 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:48.719632    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:48.821759 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:48.821720    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:49.438884 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:49.438863    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5fb545d8cf-xz62b"
Sep 11 21:53:49.548191 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:49.548157    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:50.281965 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:50.281934    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5h829"
Sep 11 21:53:50.469607 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:50.469571    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerStarted","Data":"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"}
Sep 11 21:53:50.470056 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:50.469650    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerStarted","Data":"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"}
Sep 11 21:53:50.470056 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:50.469692    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerStarted","Data":"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"}
Sep 11 21:53:50.470056 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:50.469734    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerStarted","Data":"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"}
Sep 11 21:53:50.470056 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:50.469776    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerStarted","Data":"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"}
Sep 11 21:53:50.470056 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:50.469813    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerStarted","Data":"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"}
Sep 11 21:53:50.502266 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:50.502201    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.486747109 podStartE2EDuration="7.502179427s" totalImagesPullingTime="3.996968745s" totalInitContainerRuntime="18.463573ms" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:50.499093639 +0000 UTC m=+97.234204975" watchObservedRunningTime="2026-09-11 21:53:50.502179427 +0000 UTC m=+97.237290774"
Sep 11 21:53:51.432786 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:51.432751    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:51.530233 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:51.530201    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:52.148619 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:52.148583    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:52.245303 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:52.245273    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:52.870169 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:52.870136    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:52.870555 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:52.870198    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:52.875697 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:52.875666    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:52.938604 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:52.938568    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c4bcd6df5-t8fsw"]
Sep 11 21:53:52.967924 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:52.967894    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8698c4577b-7rkpd"]
Sep 11 21:53:52.972835 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:52.972814    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:52.992554 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:52.992524    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8698c4577b-7rkpd"]
Sep 11 21:53:53.146278 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.146198    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-oauth-config\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.146278 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.146237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-config\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.146278 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.146256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-trusted-ca-bundle\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.146503 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.146352    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjvm\" (UniqueName: \"kubernetes.io/projected/fe24043f-ead0-4e0a-8a09-2a85c7c34904-kube-api-access-8mjvm\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.146503 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.146385    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-service-ca\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.146503 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.146412    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-oauth-serving-cert\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.146503 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.146478    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-serving-cert\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.247205 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.247175    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjvm\" (UniqueName: \"kubernetes.io/projected/fe24043f-ead0-4e0a-8a09-2a85c7c34904-kube-api-access-8mjvm\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.247205 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.247211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-service-ca\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.247516 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.247234    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-oauth-serving-cert\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.247516 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.247277    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-serving-cert\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.247516 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.247393    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-oauth-config\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.247516 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.247437    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-config\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.247516 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.247463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-trusted-ca-bundle\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.248151 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.248128    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-service-ca\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.248326 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.248277    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-config\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.248415 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.248392    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-trusted-ca-bundle\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.248543 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.248424    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-oauth-serving-cert\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.250145 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.250119    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-oauth-config\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.250257 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.250204    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-serving-cert\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.255046 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.255022    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjvm\" (UniqueName: \"kubernetes.io/projected/fe24043f-ead0-4e0a-8a09-2a85c7c34904-kube-api-access-8mjvm\") pod \"console-8698c4577b-7rkpd\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") " pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.286028 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.286004    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:53:53.389138 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.389062    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:53.487259 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.487182    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:53:53.487963 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.487935    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:53.569936 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.569916    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8698c4577b-7rkpd"]
Sep 11 21:53:53.572301 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:53:53.572276    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe24043f_ead0_4e0a_8a09_2a85c7c34904.slice/crio-5db8206bd6a0037cddf50498f773efe8c4f954ef5bd5b0cc585cb1462616aca9 WatchSource:0}: Error finding container 5db8206bd6a0037cddf50498f773efe8c4f954ef5bd5b0cc585cb1462616aca9: Status 404 returned error can't find the container with id 5db8206bd6a0037cddf50498f773efe8c4f954ef5bd5b0cc585cb1462616aca9
Sep 11 21:53:53.939425 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:53.939391    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:54.039759 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:54.039727    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:54.143298 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:54.143263    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:54.286822 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:54.286755    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:53:54.489372 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:54.489334    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8698c4577b-7rkpd" event={"ID":"fe24043f-ead0-4e0a-8a09-2a85c7c34904","Type":"ContainerStarted","Data":"1a71a9110aca1e5857b56e7eeeb52691c7a72baa48eaae0105eac11de9a4899d"}
Sep 11 21:53:54.489510 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:54.489390    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8698c4577b-7rkpd" event={"ID":"fe24043f-ead0-4e0a-8a09-2a85c7c34904","Type":"ContainerStarted","Data":"5db8206bd6a0037cddf50498f773efe8c4f954ef5bd5b0cc585cb1462616aca9"}
Sep 11 21:53:54.507118 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:54.507070    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8698c4577b-7rkpd" podStartSLOduration=2.507054767 podStartE2EDuration="2.507054767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:54.506559034 +0000 UTC m=+101.241670382" watchObservedRunningTime="2026-09-11 21:53:54.507054767 +0000 UTC m=+101.242166113"
Sep 11 21:53:57.257953 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:57.257915    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:57.359029 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:57.359001    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:57.739891 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:57.739853    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:57.794806 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:53:57.794777    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:03.287061 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:03.287020    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:54:03.287541 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:03.287080    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:54:03.291939 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:03.291916    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:54:03.532677 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:03.532638    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:54:03.583441 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:03.583371    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88cb58d6-jdbtb"]
Sep 11 21:54:03.676672 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:03.676635    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:03.727883 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:03.727843    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:03.783292 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:03.783262    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:08.695379 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.695330    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2" podUID="c4d96ddd-a046-422c-928b-564b04193b6a" containerName="registry" containerID="cri-o://db6a7edc4016bcbd54bd5c30ec9b4578d80fb7c9bf6416c76dfdd04a0815fab9" gracePeriod=30
Sep 11 21:54:08.941215 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.941193    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:54:08.975907 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.975852    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4d96ddd-a046-422c-928b-564b04193b6a-ca-trust-extracted\") pod \"c4d96ddd-a046-422c-928b-564b04193b6a\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") "
Sep 11 21:54:08.975907 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.975895    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-bound-sa-token\") pod \"c4d96ddd-a046-422c-928b-564b04193b6a\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") "
Sep 11 21:54:08.976049 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.975935    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4d96ddd-a046-422c-928b-564b04193b6a-registry-certificates\") pod \"c4d96ddd-a046-422c-928b-564b04193b6a\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") "
Sep 11 21:54:08.976049 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.975955    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfxj7\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-kube-api-access-tfxj7\") pod \"c4d96ddd-a046-422c-928b-564b04193b6a\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") "
Sep 11 21:54:08.976049 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.975978    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4d96ddd-a046-422c-928b-564b04193b6a-image-registry-private-configuration\") pod \"c4d96ddd-a046-422c-928b-564b04193b6a\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") "
Sep 11 21:54:08.976049 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.975999    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4d96ddd-a046-422c-928b-564b04193b6a-installation-pull-secrets\") pod \"c4d96ddd-a046-422c-928b-564b04193b6a\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") "
Sep 11 21:54:08.976230 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.976076    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4d96ddd-a046-422c-928b-564b04193b6a-trusted-ca\") pod \"c4d96ddd-a046-422c-928b-564b04193b6a\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") "
Sep 11 21:54:08.976230 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.976165    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls\") pod \"c4d96ddd-a046-422c-928b-564b04193b6a\" (UID: \"c4d96ddd-a046-422c-928b-564b04193b6a\") "
Sep 11 21:54:08.976547 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.976516    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d96ddd-a046-422c-928b-564b04193b6a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c4d96ddd-a046-422c-928b-564b04193b6a" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:08.976709 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.976679    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d96ddd-a046-422c-928b-564b04193b6a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c4d96ddd-a046-422c-928b-564b04193b6a" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:08.978553 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.978505    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d96ddd-a046-422c-928b-564b04193b6a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c4d96ddd-a046-422c-928b-564b04193b6a" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:08.978688 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.978665    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c4d96ddd-a046-422c-928b-564b04193b6a" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:54:08.978894 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.978869    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c4d96ddd-a046-422c-928b-564b04193b6a" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:54:08.978987 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.978966    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-kube-api-access-tfxj7" (OuterVolumeSpecName: "kube-api-access-tfxj7") pod "c4d96ddd-a046-422c-928b-564b04193b6a" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a"). InnerVolumeSpecName "kube-api-access-tfxj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:54:08.979028 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.978972    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d96ddd-a046-422c-928b-564b04193b6a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c4d96ddd-a046-422c-928b-564b04193b6a" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:08.983980 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:08.983959    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d96ddd-a046-422c-928b-564b04193b6a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c4d96ddd-a046-422c-928b-564b04193b6a" (UID: "c4d96ddd-a046-422c-928b-564b04193b6a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:54:09.021266 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.021200    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:09.077327 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.077286    2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4d96ddd-a046-422c-928b-564b04193b6a-ca-trust-extracted\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:09.077327 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.077326    2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-bound-sa-token\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:09.077478 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.077337    2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4d96ddd-a046-422c-928b-564b04193b6a-registry-certificates\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:09.077478 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.077347    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfxj7\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-kube-api-access-tfxj7\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:09.077478 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.077356    2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4d96ddd-a046-422c-928b-564b04193b6a-image-registry-private-configuration\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:09.077478 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.077364    2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4d96ddd-a046-422c-928b-564b04193b6a-installation-pull-secrets\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:09.077478 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.077373    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4d96ddd-a046-422c-928b-564b04193b6a-trusted-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:09.077478 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.077381    2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d96ddd-a046-422c-928b-564b04193b6a-registry-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:09.119995 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.119948    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:09.558819 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.558786    2574 generic.go:356] "Generic (PLEG): container finished" podID="c4d96ddd-a046-422c-928b-564b04193b6a" containerID="db6a7edc4016bcbd54bd5c30ec9b4578d80fb7c9bf6416c76dfdd04a0815fab9" exitCode=0
Sep 11 21:54:09.558981 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.558824    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2" event={"ID":"c4d96ddd-a046-422c-928b-564b04193b6a","Type":"ContainerDied","Data":"db6a7edc4016bcbd54bd5c30ec9b4578d80fb7c9bf6416c76dfdd04a0815fab9"}
Sep 11 21:54:09.558981 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.558828    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2"
Sep 11 21:54:09.558981 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.558862    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56d9f5f6cd-crns2" event={"ID":"c4d96ddd-a046-422c-928b-564b04193b6a","Type":"ContainerDied","Data":"6ae1214b151e1bb539257d3af7095ecf2a99d6f6f0c1dee1c37b810cd0dc9d42"}
Sep 11 21:54:09.558981 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.558882    2574 scope.go:117] "RemoveContainer" containerID="db6a7edc4016bcbd54bd5c30ec9b4578d80fb7c9bf6416c76dfdd04a0815fab9"
Sep 11 21:54:09.567377 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.567360    2574 scope.go:117] "RemoveContainer" containerID="db6a7edc4016bcbd54bd5c30ec9b4578d80fb7c9bf6416c76dfdd04a0815fab9"
Sep 11 21:54:09.567680 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:54:09.567660    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6a7edc4016bcbd54bd5c30ec9b4578d80fb7c9bf6416c76dfdd04a0815fab9\": container with ID starting with db6a7edc4016bcbd54bd5c30ec9b4578d80fb7c9bf6416c76dfdd04a0815fab9 not found: ID does not exist" containerID="db6a7edc4016bcbd54bd5c30ec9b4578d80fb7c9bf6416c76dfdd04a0815fab9"
Sep 11 21:54:09.567740 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.567688    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6a7edc4016bcbd54bd5c30ec9b4578d80fb7c9bf6416c76dfdd04a0815fab9"} err="failed to get container status \"db6a7edc4016bcbd54bd5c30ec9b4578d80fb7c9bf6416c76dfdd04a0815fab9\": rpc error: code = NotFound desc = could not find container \"db6a7edc4016bcbd54bd5c30ec9b4578d80fb7c9bf6416c76dfdd04a0815fab9\": container with ID starting with db6a7edc4016bcbd54bd5c30ec9b4578d80fb7c9bf6416c76dfdd04a0815fab9 not found: ID does not exist"
Sep 11 21:54:09.580682 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.580658    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56d9f5f6cd-crns2"]
Sep 11 21:54:09.583614 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.583592    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56d9f5f6cd-crns2"]
Sep 11 21:54:09.719788 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:09.719758    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4d96ddd-a046-422c-928b-564b04193b6a" path="/var/lib/kubelet/pods/c4d96ddd-a046-422c-928b-564b04193b6a/volumes"
Sep 11 21:54:16.381633 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:16.381574    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:16.447731 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:16.447662    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:18.598528 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:18.598502    2574 generic.go:356] "Generic (PLEG): container finished" podID="85b6eeb6-8682-432f-8a62-1a9f9f9ad358" containerID="75eeca2a690f770dc83d648d8e1f1a43fc44d14f0c54e8448adeda65de794c78" exitCode=0
Sep 11 21:54:18.598859 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:18.598580    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr" event={"ID":"85b6eeb6-8682-432f-8a62-1a9f9f9ad358","Type":"ContainerDied","Data":"75eeca2a690f770dc83d648d8e1f1a43fc44d14f0c54e8448adeda65de794c78"}
Sep 11 21:54:18.598911 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:18.598873    2574 scope.go:117] "RemoveContainer" containerID="75eeca2a690f770dc83d648d8e1f1a43fc44d14f0c54e8448adeda65de794c78"
Sep 11 21:54:19.508591 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.508550    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c4bcd6df5-t8fsw" podUID="7000c6aa-6a4f-4922-981b-8a5ed4b51a08" containerName="console" containerID="cri-o://5742b4d3c88714e2dd8e906f62538a9c02350d311763b8cca30321f2fc59dbcc" gracePeriod=15
Sep 11 21:54:19.604409 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.604382    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-j4dfr" event={"ID":"85b6eeb6-8682-432f-8a62-1a9f9f9ad358","Type":"ContainerStarted","Data":"6cf98846d4589b9e8b873c8ad623b8c73a72a95e6bd088dc8957c37c9cbc6a06"}
Sep 11 21:54:19.758761 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.758700    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:54:19.860687 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.860658    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-serving-cert\") pod \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") "
Sep 11 21:54:19.860828 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.860708    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-trusted-ca-bundle\") pod \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") "
Sep 11 21:54:19.860828 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.860741    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-oauth-config\") pod \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") "
Sep 11 21:54:19.860828 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.860776    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-oauth-serving-cert\") pod \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") "
Sep 11 21:54:19.860995 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.860848    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-config\") pod \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") "
Sep 11 21:54:19.860995 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.860881    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-service-ca\") pod \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") "
Sep 11 21:54:19.860995 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.860922    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z97vd\" (UniqueName: \"kubernetes.io/projected/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-kube-api-access-z97vd\") pod \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\" (UID: \"7000c6aa-6a4f-4922-981b-8a5ed4b51a08\") "
Sep 11 21:54:19.861207 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.861183    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7000c6aa-6a4f-4922-981b-8a5ed4b51a08" (UID: "7000c6aa-6a4f-4922-981b-8a5ed4b51a08"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:19.861364 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.861290    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-config" (OuterVolumeSpecName: "console-config") pod "7000c6aa-6a4f-4922-981b-8a5ed4b51a08" (UID: "7000c6aa-6a4f-4922-981b-8a5ed4b51a08"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:19.861364 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.861304    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-service-ca" (OuterVolumeSpecName: "service-ca") pod "7000c6aa-6a4f-4922-981b-8a5ed4b51a08" (UID: "7000c6aa-6a4f-4922-981b-8a5ed4b51a08"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:19.861364 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.861298    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7000c6aa-6a4f-4922-981b-8a5ed4b51a08" (UID: "7000c6aa-6a4f-4922-981b-8a5ed4b51a08"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:19.862973 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.862952    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7000c6aa-6a4f-4922-981b-8a5ed4b51a08" (UID: "7000c6aa-6a4f-4922-981b-8a5ed4b51a08"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:19.863056 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.862988    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7000c6aa-6a4f-4922-981b-8a5ed4b51a08" (UID: "7000c6aa-6a4f-4922-981b-8a5ed4b51a08"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:19.863159 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.863144    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-kube-api-access-z97vd" (OuterVolumeSpecName: "kube-api-access-z97vd") pod "7000c6aa-6a4f-4922-981b-8a5ed4b51a08" (UID: "7000c6aa-6a4f-4922-981b-8a5ed4b51a08"). InnerVolumeSpecName "kube-api-access-z97vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:54:19.962059 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.962038    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-trusted-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:19.962059 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.962056    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-oauth-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:19.962181 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.962067    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-oauth-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:19.962181 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.962076    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:19.962181 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.962084    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:19.962181 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.962091    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z97vd\" (UniqueName: \"kubernetes.io/projected/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-kube-api-access-z97vd\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:19.962181 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:19.962099    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7000c6aa-6a4f-4922-981b-8a5ed4b51a08-console-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:20.609969 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:20.609937    2574 generic.go:356] "Generic (PLEG): container finished" podID="7000c6aa-6a4f-4922-981b-8a5ed4b51a08" containerID="5742b4d3c88714e2dd8e906f62538a9c02350d311763b8cca30321f2fc59dbcc" exitCode=0
Sep 11 21:54:20.610375 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:20.609978    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4bcd6df5-t8fsw"
Sep 11 21:54:20.610375 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:20.610016    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4bcd6df5-t8fsw" event={"ID":"7000c6aa-6a4f-4922-981b-8a5ed4b51a08","Type":"ContainerDied","Data":"5742b4d3c88714e2dd8e906f62538a9c02350d311763b8cca30321f2fc59dbcc"}
Sep 11 21:54:20.610375 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:20.610058    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4bcd6df5-t8fsw" event={"ID":"7000c6aa-6a4f-4922-981b-8a5ed4b51a08","Type":"ContainerDied","Data":"e631cdb533631bab7cff5a0544dd0fe47659e4447aa660e0b1206bf6795d3912"}
Sep 11 21:54:20.610375 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:20.610082    2574 scope.go:117] "RemoveContainer" containerID="5742b4d3c88714e2dd8e906f62538a9c02350d311763b8cca30321f2fc59dbcc"
Sep 11 21:54:20.618577 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:20.618557    2574 scope.go:117] "RemoveContainer" containerID="5742b4d3c88714e2dd8e906f62538a9c02350d311763b8cca30321f2fc59dbcc"
Sep 11 21:54:20.618829 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:54:20.618810    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5742b4d3c88714e2dd8e906f62538a9c02350d311763b8cca30321f2fc59dbcc\": container with ID starting with 5742b4d3c88714e2dd8e906f62538a9c02350d311763b8cca30321f2fc59dbcc not found: ID does not exist" containerID="5742b4d3c88714e2dd8e906f62538a9c02350d311763b8cca30321f2fc59dbcc"
Sep 11 21:54:20.618877 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:20.618839    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5742b4d3c88714e2dd8e906f62538a9c02350d311763b8cca30321f2fc59dbcc"} err="failed to get container status \"5742b4d3c88714e2dd8e906f62538a9c02350d311763b8cca30321f2fc59dbcc\": rpc error: code = NotFound desc = could not find container \"5742b4d3c88714e2dd8e906f62538a9c02350d311763b8cca30321f2fc59dbcc\": container with ID starting with 5742b4d3c88714e2dd8e906f62538a9c02350d311763b8cca30321f2fc59dbcc not found: ID does not exist"
Sep 11 21:54:20.636521 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:20.636498    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c4bcd6df5-t8fsw"]
Sep 11 21:54:20.637683 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:20.637665    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c4bcd6df5-t8fsw"]
Sep 11 21:54:21.718858 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:21.718818    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7000c6aa-6a4f-4922-981b-8a5ed4b51a08" path="/var/lib/kubelet/pods/7000c6aa-6a4f-4922-981b-8a5ed4b51a08/volumes"
Sep 11 21:54:23.624412 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:23.624379    2574 generic.go:356] "Generic (PLEG): container finished" podID="64b6b508-ccca-4b2f-8e17-44de2a0b6e55" containerID="dab12bd8b8de2a0ad4e7519ba6887bd54ca57a0c508a49b968414526ad0bebf3" exitCode=0
Sep 11 21:54:23.624859 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:23.624461    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-ckcpn" event={"ID":"64b6b508-ccca-4b2f-8e17-44de2a0b6e55","Type":"ContainerDied","Data":"dab12bd8b8de2a0ad4e7519ba6887bd54ca57a0c508a49b968414526ad0bebf3"}
Sep 11 21:54:23.624923 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:23.624897    2574 scope.go:117] "RemoveContainer" containerID="dab12bd8b8de2a0ad4e7519ba6887bd54ca57a0c508a49b968414526ad0bebf3"
Sep 11 21:54:24.630170 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:24.630138    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-ckcpn" event={"ID":"64b6b508-ccca-4b2f-8e17-44de2a0b6e55","Type":"ContainerStarted","Data":"c6df323fc54582229057c1e3b6c36e8f8b8a636c3fddca86591082952367012d"}
Sep 11 21:54:28.606523 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.606461    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-88cb58d6-jdbtb" podUID="b7af72c2-db69-45aa-ba08-b8115ee30163" containerName="console" containerID="cri-o://f749669cb4fc10e1c394b3fc2bdb4a3d52b41b86b25f83a281836aeaa61deebf" gracePeriod=15
Sep 11 21:54:28.842012 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.841988    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:54:28.932926 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.932859    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-console-config\") pod \"b7af72c2-db69-45aa-ba08-b8115ee30163\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") "
Sep 11 21:54:28.932926 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.932897    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-oauth-serving-cert\") pod \"b7af72c2-db69-45aa-ba08-b8115ee30163\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") "
Sep 11 21:54:28.932926 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.932918    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7af72c2-db69-45aa-ba08-b8115ee30163-console-oauth-config\") pod \"b7af72c2-db69-45aa-ba08-b8115ee30163\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") "
Sep 11 21:54:28.933173 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.932981    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7af72c2-db69-45aa-ba08-b8115ee30163-console-serving-cert\") pod \"b7af72c2-db69-45aa-ba08-b8115ee30163\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") "
Sep 11 21:54:28.933173 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.933020    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-service-ca\") pod \"b7af72c2-db69-45aa-ba08-b8115ee30163\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") "
Sep 11 21:54:28.933173 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.933053    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcj58\" (UniqueName: \"kubernetes.io/projected/b7af72c2-db69-45aa-ba08-b8115ee30163-kube-api-access-gcj58\") pod \"b7af72c2-db69-45aa-ba08-b8115ee30163\" (UID: \"b7af72c2-db69-45aa-ba08-b8115ee30163\") "
Sep 11 21:54:28.933344 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.933290    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-console-config" (OuterVolumeSpecName: "console-config") pod "b7af72c2-db69-45aa-ba08-b8115ee30163" (UID: "b7af72c2-db69-45aa-ba08-b8115ee30163"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:28.933344 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.933305    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b7af72c2-db69-45aa-ba08-b8115ee30163" (UID: "b7af72c2-db69-45aa-ba08-b8115ee30163"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:28.933459 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.933418    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-service-ca" (OuterVolumeSpecName: "service-ca") pod "b7af72c2-db69-45aa-ba08-b8115ee30163" (UID: "b7af72c2-db69-45aa-ba08-b8115ee30163"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:28.935070 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.935039    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7af72c2-db69-45aa-ba08-b8115ee30163-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b7af72c2-db69-45aa-ba08-b8115ee30163" (UID: "b7af72c2-db69-45aa-ba08-b8115ee30163"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:28.935173 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.935108    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7af72c2-db69-45aa-ba08-b8115ee30163-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b7af72c2-db69-45aa-ba08-b8115ee30163" (UID: "b7af72c2-db69-45aa-ba08-b8115ee30163"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:28.935173 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:28.935139    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7af72c2-db69-45aa-ba08-b8115ee30163-kube-api-access-gcj58" (OuterVolumeSpecName: "kube-api-access-gcj58") pod "b7af72c2-db69-45aa-ba08-b8115ee30163" (UID: "b7af72c2-db69-45aa-ba08-b8115ee30163"). InnerVolumeSpecName "kube-api-access-gcj58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:54:29.033576 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.033553    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcj58\" (UniqueName: \"kubernetes.io/projected/b7af72c2-db69-45aa-ba08-b8115ee30163-kube-api-access-gcj58\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:29.033576 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.033574    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-console-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:29.033702 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.033584    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-oauth-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:29.033702 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.033593    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7af72c2-db69-45aa-ba08-b8115ee30163-console-oauth-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:29.033702 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.033601    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7af72c2-db69-45aa-ba08-b8115ee30163-console-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:29.033702 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.033609    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7af72c2-db69-45aa-ba08-b8115ee30163-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:54:29.653607 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.653568    2574 generic.go:356] "Generic (PLEG): container finished" podID="b7af72c2-db69-45aa-ba08-b8115ee30163" containerID="f749669cb4fc10e1c394b3fc2bdb4a3d52b41b86b25f83a281836aeaa61deebf" exitCode=0
Sep 11 21:54:29.654003 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.653617    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88cb58d6-jdbtb"
Sep 11 21:54:29.654003 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.653614    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88cb58d6-jdbtb" event={"ID":"b7af72c2-db69-45aa-ba08-b8115ee30163","Type":"ContainerDied","Data":"f749669cb4fc10e1c394b3fc2bdb4a3d52b41b86b25f83a281836aeaa61deebf"}
Sep 11 21:54:29.654003 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.653726    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88cb58d6-jdbtb" event={"ID":"b7af72c2-db69-45aa-ba08-b8115ee30163","Type":"ContainerDied","Data":"9af45b9ffbbab55066d4ca2d3891ec5542d63a3ccb1d6f07667b6ec3e5ac4389"}
Sep 11 21:54:29.654003 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.653751    2574 scope.go:117] "RemoveContainer" containerID="f749669cb4fc10e1c394b3fc2bdb4a3d52b41b86b25f83a281836aeaa61deebf"
Sep 11 21:54:29.662288 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.662270    2574 scope.go:117] "RemoveContainer" containerID="f749669cb4fc10e1c394b3fc2bdb4a3d52b41b86b25f83a281836aeaa61deebf"
Sep 11 21:54:29.662588 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:54:29.662571    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f749669cb4fc10e1c394b3fc2bdb4a3d52b41b86b25f83a281836aeaa61deebf\": container with ID starting with f749669cb4fc10e1c394b3fc2bdb4a3d52b41b86b25f83a281836aeaa61deebf not found: ID does not exist" containerID="f749669cb4fc10e1c394b3fc2bdb4a3d52b41b86b25f83a281836aeaa61deebf"
Sep 11 21:54:29.662648 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.662596    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f749669cb4fc10e1c394b3fc2bdb4a3d52b41b86b25f83a281836aeaa61deebf"} err="failed to get container status \"f749669cb4fc10e1c394b3fc2bdb4a3d52b41b86b25f83a281836aeaa61deebf\": rpc error: code = NotFound desc = could not find container \"f749669cb4fc10e1c394b3fc2bdb4a3d52b41b86b25f83a281836aeaa61deebf\": container with ID starting with f749669cb4fc10e1c394b3fc2bdb4a3d52b41b86b25f83a281836aeaa61deebf not found: ID does not exist"
Sep 11 21:54:29.676411 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.676387    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88cb58d6-jdbtb"]
Sep 11 21:54:29.687982 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.687962    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-88cb58d6-jdbtb"]
Sep 11 21:54:29.718431 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:29.718410    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7af72c2-db69-45aa-ba08-b8115ee30163" path="/var/lib/kubelet/pods/b7af72c2-db69-45aa-ba08-b8115ee30163/volumes"
Sep 11 21:54:32.668723 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:32.668694    2574 generic.go:356] "Generic (PLEG): container finished" podID="90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a" containerID="a582f7a7ba9aad1feb7f5fd3471ef20ea6f56f8e3695a6040e8ece65117bb92f" exitCode=0
Sep 11 21:54:32.669051 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:32.668765    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g" event={"ID":"90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a","Type":"ContainerDied","Data":"a582f7a7ba9aad1feb7f5fd3471ef20ea6f56f8e3695a6040e8ece65117bb92f"}
Sep 11 21:54:32.669092 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:32.669080    2574 scope.go:117] "RemoveContainer" containerID="a582f7a7ba9aad1feb7f5fd3471ef20ea6f56f8e3695a6040e8ece65117bb92f"
Sep 11 21:54:33.675546 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:33.675514    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbc2g" event={"ID":"90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a","Type":"ContainerStarted","Data":"e870c5927d017d0d60f7bd5fe74c823315ad44dd7891670e6468f50a17d7e686"}
Sep 11 21:54:44.286651 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:44.286615    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:54:44.301707 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:44.301683    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:54:44.739522 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:44.739493    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:54:50.719741 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:50.719703    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:53.804073 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:53.804031    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:53.877728 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:54:53.877694    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:00.832413 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:00.832377    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:00.932428 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:00.932397    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:01.034413 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:01.034378    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:01.547333 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:01.547281    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:02.298647 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.298613    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 21:55:02.299064 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.299038    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="prometheus" containerID="cri-o://0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea" gracePeriod=600
Sep 11 21:55:02.299135 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.299074    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="config-reloader" containerID="cri-o://64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1" gracePeriod=600
Sep 11 21:55:02.299135 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.299125    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="kube-rbac-proxy-thanos" containerID="cri-o://73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25" gracePeriod=600
Sep 11 21:55:02.299240 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.299110    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="kube-rbac-proxy" containerID="cri-o://05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc" gracePeriod=600
Sep 11 21:55:02.299240 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.299224    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="thanos-sidecar" containerID="cri-o://5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301" gracePeriod=600
Sep 11 21:55:02.299240 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.299231    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="kube-rbac-proxy-web" containerID="cri-o://741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607" gracePeriod=600
Sep 11 21:55:02.542285 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.542263    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:02.707468 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707432    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-1\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707642 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707495    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/222facd6-c176-4d8b-964b-6c54974f04fc-config-out\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707642 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707523    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-0\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707642 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707553    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-serving-certs-ca-bundle\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707642 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707586    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707642 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707611    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-kubelet-serving-ca-bundle\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707923 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707646    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-thanos-prometheus-http-client-file\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707923 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707697    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-metrics-client-ca\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707923 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707722    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-web-config\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707923 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707750    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbztx\" (UniqueName: \"kubernetes.io/projected/222facd6-c176-4d8b-964b-6c54974f04fc-kube-api-access-pbztx\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707923 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707776    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-metrics-client-certs\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707923 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707812    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-config\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707923 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707851    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-grpc-tls\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707923 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707846    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:55:02.707923 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707874    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/222facd6-c176-4d8b-964b-6c54974f04fc-tls-assets\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.707923 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707903    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-tls\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.708417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707932    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-2\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.708417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707956    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:55:02.708417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.707974    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-db\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.708417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.708000    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-trusted-ca-bundle\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.708417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.708044    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-kube-rbac-proxy\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.708417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.708061    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:55:02.708417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.708074    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"222facd6-c176-4d8b-964b-6c54974f04fc\" (UID: \"222facd6-c176-4d8b-964b-6c54974f04fc\") "
Sep 11 21:55:02.708417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.708335    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:55:02.708417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.708385    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.708417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.708404    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.708417 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.708419    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.708941 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.708741    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:55:02.709033 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.708991    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:55:02.710509 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.710476    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222facd6-c176-4d8b-964b-6c54974f04fc-config-out" (OuterVolumeSpecName: "config-out") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:55:02.710509 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.710481    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:02.710861 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.710831    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222facd6-c176-4d8b-964b-6c54974f04fc-kube-api-access-pbztx" (OuterVolumeSpecName: "kube-api-access-pbztx") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "kube-api-access-pbztx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:55:02.710861 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.710848    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:55:02.711460 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.711231    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:02.711460 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.711402    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-config" (OuterVolumeSpecName: "config") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:02.711770 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.711629    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:02.712015 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.711989    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:02.712087 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.712037    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:55:02.712732 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.712702    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:02.713028 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.713009    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:02.713157 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.713143    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:02.713454 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.713439    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222facd6-c176-4d8b-964b-6c54974f04fc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:55:02.721671 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.721649    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-web-config" (OuterVolumeSpecName: "web-config") pod "222facd6-c176-4d8b-964b-6c54974f04fc" (UID: "222facd6-c176-4d8b-964b-6c54974f04fc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:02.798363 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798338    2574 generic.go:356] "Generic (PLEG): container finished" podID="222facd6-c176-4d8b-964b-6c54974f04fc" containerID="73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25" exitCode=0
Sep 11 21:55:02.798363 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798358    2574 generic.go:356] "Generic (PLEG): container finished" podID="222facd6-c176-4d8b-964b-6c54974f04fc" containerID="05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc" exitCode=0
Sep 11 21:55:02.798363 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798365    2574 generic.go:356] "Generic (PLEG): container finished" podID="222facd6-c176-4d8b-964b-6c54974f04fc" containerID="741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607" exitCode=0
Sep 11 21:55:02.798565 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798371    2574 generic.go:356] "Generic (PLEG): container finished" podID="222facd6-c176-4d8b-964b-6c54974f04fc" containerID="5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301" exitCode=0
Sep 11 21:55:02.798565 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798376    2574 generic.go:356] "Generic (PLEG): container finished" podID="222facd6-c176-4d8b-964b-6c54974f04fc" containerID="64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1" exitCode=0
Sep 11 21:55:02.798565 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798381    2574 generic.go:356] "Generic (PLEG): container finished" podID="222facd6-c176-4d8b-964b-6c54974f04fc" containerID="0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea" exitCode=0
Sep 11 21:55:02.798565 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798417    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerDied","Data":"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"}
Sep 11 21:55:02.798565 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798430    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:02.798565 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798480    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerDied","Data":"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"}
Sep 11 21:55:02.798565 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798507    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerDied","Data":"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"}
Sep 11 21:55:02.798565 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798531    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerDied","Data":"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"}
Sep 11 21:55:02.798565 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798554    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerDied","Data":"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"}
Sep 11 21:55:02.798983 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798578    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerDied","Data":"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"}
Sep 11 21:55:02.798983 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798553    2574 scope.go:117] "RemoveContainer" containerID="73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"
Sep 11 21:55:02.798983 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.798624    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"222facd6-c176-4d8b-964b-6c54974f04fc","Type":"ContainerDied","Data":"f9bf4783601393bf64ce92a81a38b5d969b37e68198186fbe61de52bf07f3a74"}
Sep 11 21:55:02.806342 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.806225    2574 scope.go:117] "RemoveContainer" containerID="05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.810909    2574 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-kube-rbac-proxy\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.810933    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.810956    2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/222facd6-c176-4d8b-964b-6c54974f04fc-config-out\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.810971    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.810985    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.810999    2574 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.811017    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-configmap-metrics-client-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.811051    2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-web-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.811067    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbztx\" (UniqueName: \"kubernetes.io/projected/222facd6-c176-4d8b-964b-6c54974f04fc-kube-api-access-pbztx\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.811080    2574 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-metrics-client-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.811097    2574 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.811110    2574 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-grpc-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.811122    2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/222facd6-c176-4d8b-964b-6c54974f04fc-tls-assets\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.811134    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/222facd6-c176-4d8b-964b-6c54974f04fc-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.811150    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.811163    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-k8s-db\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.811810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.811176    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222facd6-c176-4d8b-964b-6c54974f04fc-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:02.815805 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.815789    2574 scope.go:117] "RemoveContainer" containerID="741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"
Sep 11 21:55:02.822403 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.822381    2574 scope.go:117] "RemoveContainer" containerID="5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"
Sep 11 21:55:02.826041 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.826011    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 21:55:02.829508 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.829493    2574 scope.go:117] "RemoveContainer" containerID="64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"
Sep 11 21:55:02.832900 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.832878    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 21:55:02.836412 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.836391    2574 scope.go:117] "RemoveContainer" containerID="0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"
Sep 11 21:55:02.843351 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.843329    2574 scope.go:117] "RemoveContainer" containerID="20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"
Sep 11 21:55:02.849791 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.849775    2574 scope.go:117] "RemoveContainer" containerID="73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"
Sep 11 21:55:02.850050 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:55:02.850033    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25\": container with ID starting with 73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25 not found: ID does not exist" containerID="73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"
Sep 11 21:55:02.850116 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.850059    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"} err="failed to get container status \"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25\": rpc error: code = NotFound desc = could not find container \"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25\": container with ID starting with 73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25 not found: ID does not exist"
Sep 11 21:55:02.850116 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.850077    2574 scope.go:117] "RemoveContainer" containerID="05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"
Sep 11 21:55:02.850332 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:55:02.850296    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc\": container with ID starting with 05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc not found: ID does not exist" containerID="05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"
Sep 11 21:55:02.850432 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.850341    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"} err="failed to get container status \"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc\": rpc error: code = NotFound desc = could not find container \"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc\": container with ID starting with 05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc not found: ID does not exist"
Sep 11 21:55:02.850432 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.850363    2574 scope.go:117] "RemoveContainer" containerID="741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"
Sep 11 21:55:02.850603 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:55:02.850587    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607\": container with ID starting with 741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607 not found: ID does not exist" containerID="741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"
Sep 11 21:55:02.850648 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.850606    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"} err="failed to get container status \"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607\": rpc error: code = NotFound desc = could not find container \"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607\": container with ID starting with 741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607 not found: ID does not exist"
Sep 11 21:55:02.850648 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.850619    2574 scope.go:117] "RemoveContainer" containerID="5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"
Sep 11 21:55:02.850842 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:55:02.850828    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301\": container with ID starting with 5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301 not found: ID does not exist" containerID="5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"
Sep 11 21:55:02.850883 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.850845    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"} err="failed to get container status \"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301\": rpc error: code = NotFound desc = could not find container \"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301\": container with ID starting with 5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301 not found: ID does not exist"
Sep 11 21:55:02.850883 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.850856    2574 scope.go:117] "RemoveContainer" containerID="64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"
Sep 11 21:55:02.851054 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:55:02.851038    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1\": container with ID starting with 64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1 not found: ID does not exist" containerID="64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"
Sep 11 21:55:02.851098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.851057    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"} err="failed to get container status \"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1\": rpc error: code = NotFound desc = could not find container \"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1\": container with ID starting with 64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1 not found: ID does not exist"
Sep 11 21:55:02.851098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.851070    2574 scope.go:117] "RemoveContainer" containerID="0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"
Sep 11 21:55:02.851346 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:55:02.851299    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea\": container with ID starting with 0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea not found: ID does not exist" containerID="0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"
Sep 11 21:55:02.851458 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.851345    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"} err="failed to get container status \"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea\": rpc error: code = NotFound desc = could not find container \"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea\": container with ID starting with 0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea not found: ID does not exist"
Sep 11 21:55:02.851458 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.851397    2574 scope.go:117] "RemoveContainer" containerID="20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"
Sep 11 21:55:02.851682 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:55:02.851658    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44\": container with ID starting with 20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44 not found: ID does not exist" containerID="20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"
Sep 11 21:55:02.851727 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.851688    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"} err="failed to get container status \"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44\": rpc error: code = NotFound desc = could not find container \"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44\": container with ID starting with 20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44 not found: ID does not exist"
Sep 11 21:55:02.851727 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.851702    2574 scope.go:117] "RemoveContainer" containerID="73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"
Sep 11 21:55:02.852016 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.851993    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"} err="failed to get container status \"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25\": rpc error: code = NotFound desc = could not find container \"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25\": container with ID starting with 73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25 not found: ID does not exist"
Sep 11 21:55:02.852071 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.852016    2574 scope.go:117] "RemoveContainer" containerID="05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"
Sep 11 21:55:02.852247 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.852228    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"} err="failed to get container status \"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc\": rpc error: code = NotFound desc = could not find container \"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc\": container with ID starting with 05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc not found: ID does not exist"
Sep 11 21:55:02.852292 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.852249    2574 scope.go:117] "RemoveContainer" containerID="741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"
Sep 11 21:55:02.852485 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.852469    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"} err="failed to get container status \"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607\": rpc error: code = NotFound desc = could not find container \"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607\": container with ID starting with 741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607 not found: ID does not exist"
Sep 11 21:55:02.852522 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.852485    2574 scope.go:117] "RemoveContainer" containerID="5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"
Sep 11 21:55:02.852706 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.852690    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"} err="failed to get container status \"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301\": rpc error: code = NotFound desc = could not find container \"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301\": container with ID starting with 5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301 not found: ID does not exist"
Sep 11 21:55:02.852743 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.852707    2574 scope.go:117] "RemoveContainer" containerID="64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"
Sep 11 21:55:02.852959 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.852941    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"} err="failed to get container status \"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1\": rpc error: code = NotFound desc = could not find container \"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1\": container with ID starting with 64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1 not found: ID does not exist"
Sep 11 21:55:02.853006 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.852961    2574 scope.go:117] "RemoveContainer" containerID="0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"
Sep 11 21:55:02.853197 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.853168    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"} err="failed to get container status \"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea\": rpc error: code = NotFound desc = could not find container \"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea\": container with ID starting with 0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea not found: ID does not exist"
Sep 11 21:55:02.853197 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.853189    2574 scope.go:117] "RemoveContainer" containerID="20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"
Sep 11 21:55:02.853431 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.853403    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"} err="failed to get container status \"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44\": rpc error: code = NotFound desc = could not find container \"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44\": container with ID starting with 20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44 not found: ID does not exist"
Sep 11 21:55:02.853502 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.853434    2574 scope.go:117] "RemoveContainer" containerID="73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"
Sep 11 21:55:02.853646 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.853628    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"} err="failed to get container status \"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25\": rpc error: code = NotFound desc = could not find container \"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25\": container with ID starting with 73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25 not found: ID does not exist"
Sep 11 21:55:02.853692 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.853646    2574 scope.go:117] "RemoveContainer" containerID="05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"
Sep 11 21:55:02.853839 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.853825    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"} err="failed to get container status \"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc\": rpc error: code = NotFound desc = could not find container \"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc\": container with ID starting with 05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc not found: ID does not exist"
Sep 11 21:55:02.853876 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.853839    2574 scope.go:117] "RemoveContainer" containerID="741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"
Sep 11 21:55:02.854042 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.854023    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"} err="failed to get container status \"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607\": rpc error: code = NotFound desc = could not find container \"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607\": container with ID starting with 741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607 not found: ID does not exist"
Sep 11 21:55:02.854086 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.854044    2574 scope.go:117] "RemoveContainer" containerID="5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"
Sep 11 21:55:02.854248 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.854233    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"} err="failed to get container status \"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301\": rpc error: code = NotFound desc = could not find container \"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301\": container with ID starting with 5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301 not found: ID does not exist"
Sep 11 21:55:02.854248 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.854247    2574 scope.go:117] "RemoveContainer" containerID="64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"
Sep 11 21:55:02.854525 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.854506    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"} err="failed to get container status \"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1\": rpc error: code = NotFound desc = could not find container \"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1\": container with ID starting with 64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1 not found: ID does not exist"
Sep 11 21:55:02.854574 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.854526    2574 scope.go:117] "RemoveContainer" containerID="0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"
Sep 11 21:55:02.854754 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.854732    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"} err="failed to get container status \"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea\": rpc error: code = NotFound desc = could not find container \"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea\": container with ID starting with 0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea not found: ID does not exist"
Sep 11 21:55:02.854754 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.854753    2574 scope.go:117] "RemoveContainer" containerID="20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"
Sep 11 21:55:02.854969 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.854953    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"} err="failed to get container status \"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44\": rpc error: code = NotFound desc = could not find container \"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44\": container with ID starting with 20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44 not found: ID does not exist"
Sep 11 21:55:02.854969 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.854968    2574 scope.go:117] "RemoveContainer" containerID="73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"
Sep 11 21:55:02.855203 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.855183    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"} err="failed to get container status \"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25\": rpc error: code = NotFound desc = could not find container \"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25\": container with ID starting with 73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25 not found: ID does not exist"
Sep 11 21:55:02.855266 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.855204    2574 scope.go:117] "RemoveContainer" containerID="05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"
Sep 11 21:55:02.855442 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.855425    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"} err="failed to get container status \"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc\": rpc error: code = NotFound desc = could not find container \"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc\": container with ID starting with 05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc not found: ID does not exist"
Sep 11 21:55:02.855505 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.855442    2574 scope.go:117] "RemoveContainer" containerID="741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"
Sep 11 21:55:02.855662 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.855645    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"} err="failed to get container status \"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607\": rpc error: code = NotFound desc = could not find container \"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607\": container with ID starting with 741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607 not found: ID does not exist"
Sep 11 21:55:02.855715 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.855663    2574 scope.go:117] "RemoveContainer" containerID="5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"
Sep 11 21:55:02.855897 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.855879    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"} err="failed to get container status \"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301\": rpc error: code = NotFound desc = could not find container \"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301\": container with ID starting with 5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301 not found: ID does not exist"
Sep 11 21:55:02.855966 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.855899    2574 scope.go:117] "RemoveContainer" containerID="64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"
Sep 11 21:55:02.856132 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.856116    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"} err="failed to get container status \"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1\": rpc error: code = NotFound desc = could not find container \"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1\": container with ID starting with 64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1 not found: ID does not exist"
Sep 11 21:55:02.856181 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.856132    2574 scope.go:117] "RemoveContainer" containerID="0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"
Sep 11 21:55:02.856362 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.856343    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"} err="failed to get container status \"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea\": rpc error: code = NotFound desc = could not find container \"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea\": container with ID starting with 0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea not found: ID does not exist"
Sep 11 21:55:02.856403 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.856362    2574 scope.go:117] "RemoveContainer" containerID="20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"
Sep 11 21:55:02.856567 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.856552    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"} err="failed to get container status \"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44\": rpc error: code = NotFound desc = could not find container \"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44\": container with ID starting with 20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44 not found: ID does not exist"
Sep 11 21:55:02.856606 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.856567    2574 scope.go:117] "RemoveContainer" containerID="73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"
Sep 11 21:55:02.856776 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.856760    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"} err="failed to get container status \"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25\": rpc error: code = NotFound desc = could not find container \"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25\": container with ID starting with 73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25 not found: ID does not exist"
Sep 11 21:55:02.856820 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.856776    2574 scope.go:117] "RemoveContainer" containerID="05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"
Sep 11 21:55:02.856963 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.856948    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"} err="failed to get container status \"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc\": rpc error: code = NotFound desc = could not find container \"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc\": container with ID starting with 05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc not found: ID does not exist"
Sep 11 21:55:02.857003 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.856964    2574 scope.go:117] "RemoveContainer" containerID="741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"
Sep 11 21:55:02.857150 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.857133    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"} err="failed to get container status \"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607\": rpc error: code = NotFound desc = could not find container \"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607\": container with ID starting with 741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607 not found: ID does not exist"
Sep 11 21:55:02.857195 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.857151    2574 scope.go:117] "RemoveContainer" containerID="5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"
Sep 11 21:55:02.857364 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.857346    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"} err="failed to get container status \"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301\": rpc error: code = NotFound desc = could not find container \"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301\": container with ID starting with 5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301 not found: ID does not exist"
Sep 11 21:55:02.857411 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.857365    2574 scope.go:117] "RemoveContainer" containerID="64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"
Sep 11 21:55:02.857569 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.857554    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"} err="failed to get container status \"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1\": rpc error: code = NotFound desc = could not find container \"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1\": container with ID starting with 64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1 not found: ID does not exist"
Sep 11 21:55:02.857612 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.857569    2574 scope.go:117] "RemoveContainer" containerID="0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"
Sep 11 21:55:02.857773 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.857755    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"} err="failed to get container status \"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea\": rpc error: code = NotFound desc = could not find container \"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea\": container with ID starting with 0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea not found: ID does not exist"
Sep 11 21:55:02.857822 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.857774    2574 scope.go:117] "RemoveContainer" containerID="20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"
Sep 11 21:55:02.857983 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.857967    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"} err="failed to get container status \"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44\": rpc error: code = NotFound desc = could not find container \"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44\": container with ID starting with 20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44 not found: ID does not exist"
Sep 11 21:55:02.858019 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.857984    2574 scope.go:117] "RemoveContainer" containerID="73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"
Sep 11 21:55:02.858195 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.858177    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25"} err="failed to get container status \"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25\": rpc error: code = NotFound desc = could not find container \"73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25\": container with ID starting with 73fb68791eb8eb47fee4c01027b7947eacb1e83d166ee9e0c7bc67d814de5b25 not found: ID does not exist"
Sep 11 21:55:02.858238 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.858195    2574 scope.go:117] "RemoveContainer" containerID="05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"
Sep 11 21:55:02.858398 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.858382    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc"} err="failed to get container status \"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc\": rpc error: code = NotFound desc = could not find container \"05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc\": container with ID starting with 05d1ce61c94917684b15df1abe1e5c022e898570dbc5caa4b4a95684c3f91afc not found: ID does not exist"
Sep 11 21:55:02.858398 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.858397    2574 scope.go:117] "RemoveContainer" containerID="741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"
Sep 11 21:55:02.858625 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.858599    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607"} err="failed to get container status \"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607\": rpc error: code = NotFound desc = could not find container \"741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607\": container with ID starting with 741d3197df9db78929d1d0520a5e3e26f3c0f0230c28e20e547961b493b23607 not found: ID does not exist"
Sep 11 21:55:02.858672 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.858625    2574 scope.go:117] "RemoveContainer" containerID="5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"
Sep 11 21:55:02.858827 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.858806    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301"} err="failed to get container status \"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301\": rpc error: code = NotFound desc = could not find container \"5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301\": container with ID starting with 5eae542f5a1f7b3c9dffeeb8231eaf130ad0b0471f81e5390ab7c2dea8fa7301 not found: ID does not exist"
Sep 11 21:55:02.858827 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.858826    2574 scope.go:117] "RemoveContainer" containerID="64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"
Sep 11 21:55:02.859046 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.859026    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1"} err="failed to get container status \"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1\": rpc error: code = NotFound desc = could not find container \"64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1\": container with ID starting with 64b9aa5b7efb663004079e138573c50a0152d6fbeef476c6afe5f56adb0dc7c1 not found: ID does not exist"
Sep 11 21:55:02.859099 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.859046    2574 scope.go:117] "RemoveContainer" containerID="0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"
Sep 11 21:55:02.859275 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.859258    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea"} err="failed to get container status \"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea\": rpc error: code = NotFound desc = could not find container \"0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea\": container with ID starting with 0b9804af7c184c3efc88a2b7f8db4f6ea17b380afaf053477ca94c4f9f1a14ea not found: ID does not exist"
Sep 11 21:55:02.859275 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.859274    2574 scope.go:117] "RemoveContainer" containerID="20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"
Sep 11 21:55:02.859513 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.859493    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44"} err="failed to get container status \"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44\": rpc error: code = NotFound desc = could not find container \"20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44\": container with ID starting with 20d0f66ce1bcad8276e74357c59ad0eb48631fff4c5b505fbd1af8102a4d4a44 not found: ID does not exist"
Sep 11 21:55:02.861364 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861346    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 21:55:02.861692 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861677    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="kube-rbac-proxy-thanos"
Sep 11 21:55:02.861765 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861695    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="kube-rbac-proxy-thanos"
Sep 11 21:55:02.861765 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861712    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="kube-rbac-proxy"
Sep 11 21:55:02.861765 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861721    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="kube-rbac-proxy"
Sep 11 21:55:02.861765 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861736    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="thanos-sidecar"
Sep 11 21:55:02.861765 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861745    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="thanos-sidecar"
Sep 11 21:55:02.861765 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861755    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7000c6aa-6a4f-4922-981b-8a5ed4b51a08" containerName="console"
Sep 11 21:55:02.861765 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861763    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7000c6aa-6a4f-4922-981b-8a5ed4b51a08" containerName="console"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861774    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7af72c2-db69-45aa-ba08-b8115ee30163" containerName="console"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861783    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7af72c2-db69-45aa-ba08-b8115ee30163" containerName="console"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861792    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="kube-rbac-proxy-web"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861800    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="kube-rbac-proxy-web"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861812    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d96ddd-a046-422c-928b-564b04193b6a" containerName="registry"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861820    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d96ddd-a046-422c-928b-564b04193b6a" containerName="registry"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861839    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="config-reloader"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861847    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="config-reloader"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861859    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="init-config-reloader"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861868    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="init-config-reloader"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861877    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="prometheus"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861885    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="prometheus"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861958    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7af72c2-db69-45aa-ba08-b8115ee30163" containerName="console"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861969    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="prometheus"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861978    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="kube-rbac-proxy-web"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861986    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="kube-rbac-proxy-thanos"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.861996    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="config-reloader"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.862006    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d96ddd-a046-422c-928b-564b04193b6a" containerName="registry"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.862014    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="kube-rbac-proxy"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.862023    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" containerName="thanos-sidecar"
Sep 11 21:55:02.862095 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.862034    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7000c6aa-6a4f-4922-981b-8a5ed4b51a08" containerName="console"
Sep 11 21:55:02.867847 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.867831    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:02.870789 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.870739    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 21:55:02.870789 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.870772    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 21:55:02.870968 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.870772    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 21:55:02.870968 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.870840    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jzhhc\""
Sep 11 21:55:02.871090 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.871077    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 21:55:02.871400 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.871385    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 21:55:02.871540 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.871529    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 21:55:02.873612 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.873589    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 21:55:02.873749 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.873596    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d4lfk5i11iqhs\""
Sep 11 21:55:02.873749 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.873719    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 21:55:02.873884 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.873807    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 21:55:02.873964 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.873890    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 21:55:02.874038 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.873989    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 21:55:02.874117 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.874102    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 21:55:02.874208 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.874191    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 21:55:02.876013 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.875995    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 21:55:02.877688 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.877670    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 21:55:02.883559 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:02.883539    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 21:55:03.013146 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013067    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-web-config\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013146 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013106    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013146 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013131    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013401 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013206    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013401 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013246    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013401 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-config\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013401 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013336    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013401 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013393    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrnzq\" (UniqueName: \"kubernetes.io/projected/18f95c61-356a-4dc0-be36-13c25a5c5a0c-kube-api-access-xrnzq\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013593 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013419    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013593 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013448    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013593 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013469    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18f95c61-356a-4dc0-be36-13c25a5c5a0c-config-out\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013593 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013593 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013501    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013593 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013518    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013783 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013598    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013783 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013616    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013783 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013632    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013783 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013645    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013783 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013684    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.013783 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.013707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18f95c61-356a-4dc0-be36-13c25a5c5a0c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.114686 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.114654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.114849 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.114694    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18f95c61-356a-4dc0-be36-13c25a5c5a0c-config-out\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.114849 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.114814    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.114849 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.114847    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115017 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.114874    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115017 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.114910    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115017 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.114938    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115017 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.114962    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115017 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.114988    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115017 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.115019    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115330 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.115048    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18f95c61-356a-4dc0-be36-13c25a5c5a0c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115330 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.115084    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-web-config\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115330 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.115123    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115330 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.115149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115330 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.115185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115330 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.115220    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115330 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.115248    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-config\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115682 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.115302    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115682 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.115376    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrnzq\" (UniqueName: \"kubernetes.io/projected/18f95c61-356a-4dc0-be36-13c25a5c5a0c-kube-api-access-xrnzq\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115682 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.115377    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115682 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.115408    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.115682 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.115619    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.116514 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.116461    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.116730 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.116644    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.116881 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.116851    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.117834 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.117806    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.117935 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.117835    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.117935 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.117918    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.118061 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.118032    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.118160 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.118131    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.118240 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.118225    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18f95c61-356a-4dc0-be36-13c25a5c5a0c-config-out\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.119650 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.119618    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.119820 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.119793    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-config\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.120082 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.120063    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.120129 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.120097    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.120213 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.120193    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18f95c61-356a-4dc0-be36-13c25a5c5a0c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.120274 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.120260    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.120769 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.120753    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18f95c61-356a-4dc0-be36-13c25a5c5a0c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.120937 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.120922    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18f95c61-356a-4dc0-be36-13c25a5c5a0c-web-config\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.124843 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.124823    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrnzq\" (UniqueName: \"kubernetes.io/projected/18f95c61-356a-4dc0-be36-13c25a5c5a0c-kube-api-access-xrnzq\") pod \"prometheus-k8s-0\" (UID: \"18f95c61-356a-4dc0-be36-13c25a5c5a0c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.179772 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.179750    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:03.460084 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:55:03.460054    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f95c61_356a_4dc0_be36_13c25a5c5a0c.slice/crio-1f89cb7d79c65f832aa1f376a9c30a3740002c5f5c141f44012cb30be52c7a51 WatchSource:0}: Error finding container 1f89cb7d79c65f832aa1f376a9c30a3740002c5f5c141f44012cb30be52c7a51: Status 404 returned error can't find the container with id 1f89cb7d79c65f832aa1f376a9c30a3740002c5f5c141f44012cb30be52c7a51
Sep 11 21:55:03.461120 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.461099    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 21:55:03.719708 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.719677    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="222facd6-c176-4d8b-964b-6c54974f04fc" path="/var/lib/kubelet/pods/222facd6-c176-4d8b-964b-6c54974f04fc/volumes"
Sep 11 21:55:03.804371 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.804346    2574 generic.go:356] "Generic (PLEG): container finished" podID="18f95c61-356a-4dc0-be36-13c25a5c5a0c" containerID="c97338e9dbf1781a54a2ee9cd3e1655a76ec779128c31560fee7c37415ee87b9" exitCode=0
Sep 11 21:55:03.804492 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.804427    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18f95c61-356a-4dc0-be36-13c25a5c5a0c","Type":"ContainerDied","Data":"c97338e9dbf1781a54a2ee9cd3e1655a76ec779128c31560fee7c37415ee87b9"}
Sep 11 21:55:03.804538 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:03.804497    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18f95c61-356a-4dc0-be36-13c25a5c5a0c","Type":"ContainerStarted","Data":"1f89cb7d79c65f832aa1f376a9c30a3740002c5f5c141f44012cb30be52c7a51"}
Sep 11 21:55:04.813057 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:04.813019    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18f95c61-356a-4dc0-be36-13c25a5c5a0c","Type":"ContainerStarted","Data":"29f2044b14388b9a783bff9d89c51eab2ffb5dec4bff79dab5b1f5f9d6f27e6a"}
Sep 11 21:55:04.813408 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:04.813084    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18f95c61-356a-4dc0-be36-13c25a5c5a0c","Type":"ContainerStarted","Data":"1a50ee325a89a0e09ef45f2e803fa22ebca4c5e81ad3d8e9ccfe320354a96acb"}
Sep 11 21:55:04.813408 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:04.813112    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18f95c61-356a-4dc0-be36-13c25a5c5a0c","Type":"ContainerStarted","Data":"25ac67d129b759cbce0918b336bdfe8ba07a76910f4d5b16707dbf690fdb2ff9"}
Sep 11 21:55:04.813408 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:04.813137    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18f95c61-356a-4dc0-be36-13c25a5c5a0c","Type":"ContainerStarted","Data":"dd129712d1289ed1bab48290e8c7c46a7d84126e95577a964c2e915efc3ce5c7"}
Sep 11 21:55:04.813408 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:04.813160    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18f95c61-356a-4dc0-be36-13c25a5c5a0c","Type":"ContainerStarted","Data":"3215cb1882c08702651f07ab0482ed7ed2587b51e41ccf8c1a3520468489a286"}
Sep 11 21:55:04.813408 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:04.813182    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18f95c61-356a-4dc0-be36-13c25a5c5a0c","Type":"ContainerStarted","Data":"f03de0c5dfca47a0c52f79e5a5b5cb1231d99b4c6d973da206eac7688ea1e96f"}
Sep 11 21:55:04.843947 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:04.843903    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8251128100000003 podStartE2EDuration="2.843886473s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.773663ms" isStatefulPod=false podCreationTimestamp="2026-09-11 21:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:55:04.841396081 +0000 UTC m=+171.576507427" watchObservedRunningTime="2026-09-11 21:55:04.843886473 +0000 UTC m=+171.578997822"
Sep 11 21:55:05.623293 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:05.623256    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:05.726530 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:05.726473    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:08.180635 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:08.180594    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:55:08.995174 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:08.995139    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:09.096206 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:09.096162    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:11.155772 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:11.155742    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8698c4577b-7rkpd"]
Sep 11 21:55:11.258804 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:11.258767    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:11.373395 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:11.373358    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:11.467543 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:11.467476    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:33.688684 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:33.688645    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:33.791536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:33.791505    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:36.178149 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.178091    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8698c4577b-7rkpd" podUID="fe24043f-ead0-4e0a-8a09-2a85c7c34904" containerName="console" containerID="cri-o://1a71a9110aca1e5857b56e7eeeb52691c7a72baa48eaae0105eac11de9a4899d" gracePeriod=15
Sep 11 21:55:36.413277 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.413252    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:55:36.469075 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.469010    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-oauth-config\") pod \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") "
Sep 11 21:55:36.469194 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.469098    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-service-ca\") pod \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") "
Sep 11 21:55:36.469194 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.469141    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-serving-cert\") pod \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") "
Sep 11 21:55:36.469194 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.469173    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-oauth-serving-cert\") pod \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") "
Sep 11 21:55:36.469369 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.469216    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-trusted-ca-bundle\") pod \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") "
Sep 11 21:55:36.469369 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.469244    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mjvm\" (UniqueName: \"kubernetes.io/projected/fe24043f-ead0-4e0a-8a09-2a85c7c34904-kube-api-access-8mjvm\") pod \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") "
Sep 11 21:55:36.469369 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.469272    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-config\") pod \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\" (UID: \"fe24043f-ead0-4e0a-8a09-2a85c7c34904\") "
Sep 11 21:55:36.469676 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.469549    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-service-ca" (OuterVolumeSpecName: "service-ca") pod "fe24043f-ead0-4e0a-8a09-2a85c7c34904" (UID: "fe24043f-ead0-4e0a-8a09-2a85c7c34904"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:55:36.469781 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.469708    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fe24043f-ead0-4e0a-8a09-2a85c7c34904" (UID: "fe24043f-ead0-4e0a-8a09-2a85c7c34904"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:55:36.469854 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.469805    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-config" (OuterVolumeSpecName: "console-config") pod "fe24043f-ead0-4e0a-8a09-2a85c7c34904" (UID: "fe24043f-ead0-4e0a-8a09-2a85c7c34904"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:55:36.469907 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.469843    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fe24043f-ead0-4e0a-8a09-2a85c7c34904" (UID: "fe24043f-ead0-4e0a-8a09-2a85c7c34904"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:55:36.471279 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.471254    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fe24043f-ead0-4e0a-8a09-2a85c7c34904" (UID: "fe24043f-ead0-4e0a-8a09-2a85c7c34904"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:36.471279 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.471270    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fe24043f-ead0-4e0a-8a09-2a85c7c34904" (UID: "fe24043f-ead0-4e0a-8a09-2a85c7c34904"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:36.471452 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.471426    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe24043f-ead0-4e0a-8a09-2a85c7c34904-kube-api-access-8mjvm" (OuterVolumeSpecName: "kube-api-access-8mjvm") pod "fe24043f-ead0-4e0a-8a09-2a85c7c34904" (UID: "fe24043f-ead0-4e0a-8a09-2a85c7c34904"). InnerVolumeSpecName "kube-api-access-8mjvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:55:36.570140 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.570106    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-oauth-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:36.570140 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.570138    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:36.570140 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.570147    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:36.570327 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.570157    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-oauth-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:36.570327 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.570166    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-trusted-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:36.570327 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.570174    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mjvm\" (UniqueName: \"kubernetes.io/projected/fe24043f-ead0-4e0a-8a09-2a85c7c34904-kube-api-access-8mjvm\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:36.570327 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.570184    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe24043f-ead0-4e0a-8a09-2a85c7c34904-console-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 21:55:36.944652 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.944624    2574 generic.go:356] "Generic (PLEG): container finished" podID="fe24043f-ead0-4e0a-8a09-2a85c7c34904" containerID="1a71a9110aca1e5857b56e7eeeb52691c7a72baa48eaae0105eac11de9a4899d" exitCode=0
Sep 11 21:55:36.944803 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.944673    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8698c4577b-7rkpd"
Sep 11 21:55:36.944803 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.944712    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8698c4577b-7rkpd" event={"ID":"fe24043f-ead0-4e0a-8a09-2a85c7c34904","Type":"ContainerDied","Data":"1a71a9110aca1e5857b56e7eeeb52691c7a72baa48eaae0105eac11de9a4899d"}
Sep 11 21:55:36.944803 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.944763    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8698c4577b-7rkpd" event={"ID":"fe24043f-ead0-4e0a-8a09-2a85c7c34904","Type":"ContainerDied","Data":"5db8206bd6a0037cddf50498f773efe8c4f954ef5bd5b0cc585cb1462616aca9"}
Sep 11 21:55:36.944803 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.944784    2574 scope.go:117] "RemoveContainer" containerID="1a71a9110aca1e5857b56e7eeeb52691c7a72baa48eaae0105eac11de9a4899d"
Sep 11 21:55:36.953090 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.953075    2574 scope.go:117] "RemoveContainer" containerID="1a71a9110aca1e5857b56e7eeeb52691c7a72baa48eaae0105eac11de9a4899d"
Sep 11 21:55:36.953379 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:55:36.953359    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a71a9110aca1e5857b56e7eeeb52691c7a72baa48eaae0105eac11de9a4899d\": container with ID starting with 1a71a9110aca1e5857b56e7eeeb52691c7a72baa48eaae0105eac11de9a4899d not found: ID does not exist" containerID="1a71a9110aca1e5857b56e7eeeb52691c7a72baa48eaae0105eac11de9a4899d"
Sep 11 21:55:36.953485 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.953388    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a71a9110aca1e5857b56e7eeeb52691c7a72baa48eaae0105eac11de9a4899d"} err="failed to get container status \"1a71a9110aca1e5857b56e7eeeb52691c7a72baa48eaae0105eac11de9a4899d\": rpc error: code = NotFound desc = could not find container \"1a71a9110aca1e5857b56e7eeeb52691c7a72baa48eaae0105eac11de9a4899d\": container with ID starting with 1a71a9110aca1e5857b56e7eeeb52691c7a72baa48eaae0105eac11de9a4899d not found: ID does not exist"
Sep 11 21:55:36.965950 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.965929    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8698c4577b-7rkpd"]
Sep 11 21:55:36.969815 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:36.969794    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8698c4577b-7rkpd"]
Sep 11 21:55:37.719195 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:37.719158    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe24043f-ead0-4e0a-8a09-2a85c7c34904" path="/var/lib/kubelet/pods/fe24043f-ead0-4e0a-8a09-2a85c7c34904/volumes"
Sep 11 21:55:54.591216 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:54.591168    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:54.653705 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:55:54.653674    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:00.867253 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:00.867222    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:02.246636 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:02.246590    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:03.180866 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:03.180831    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:56:03.196411 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:03.196382    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:56:04.062715 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:04.062689    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:56:04.492349 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:04.492287    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:04.591866 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:04.591835    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:04.705776 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:04.705742    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:05.847075 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:05.847017    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:05.947684 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:05.947648    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:06.383393 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:06.383365    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:06.502271 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:06.502236    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:08.119988 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:08.119958    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:08.258335 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:08.258286    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:11.270503 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:11.270469    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:11.337265 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:11.337236    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:15.399059 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:15.399028    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:15.470639 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:15.470607    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:16.066488 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:16.066455    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:18.790108 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:18.790078    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:18.891300 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:18.891269    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:24.781767 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:24.781723    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:24.883162 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:24.883125    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:26.544396 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:26.544350    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:26.609369 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:26.609335    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:26.792114 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:26.792061    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:42.551584 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:42.551548    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:42.650263 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:42.650229    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:43.182037 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.182001    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:43.559562 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.559474    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:43.667848 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.667809    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:43.732403 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.732373    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"]
Sep 11 21:56:43.732773 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.732754    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe24043f-ead0-4e0a-8a09-2a85c7c34904" containerName="console"
Sep 11 21:56:43.732872 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.732776    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe24043f-ead0-4e0a-8a09-2a85c7c34904" containerName="console"
Sep 11 21:56:43.732872 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.732849    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe24043f-ead0-4e0a-8a09-2a85c7c34904" containerName="console"
Sep 11 21:56:43.735887 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.735865    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.739090 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.739061    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 21:56:43.739223 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.739155    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 21:56:43.739356 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.739298    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bfn6m\""
Sep 11 21:56:43.739433 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.739403    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 21:56:43.740944 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.740920    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 21:56:43.794296 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.794263    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecc7b580-81a5-404f-a8f9-3cd655b40a23-tmp\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.794499 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.794328    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ecc7b580-81a5-404f-a8f9-3cd655b40a23-manifests\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.794499 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.794411    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7cr2\" (UniqueName: \"kubernetes.io/projected/ecc7b580-81a5-404f-a8f9-3cd655b40a23-kube-api-access-k7cr2\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.794620 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.794602    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ecc7b580-81a5-404f-a8f9-3cd655b40a23-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.794742 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.794636    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecc7b580-81a5-404f-a8f9-3cd655b40a23-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.801512 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.801488    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"]
Sep 11 21:56:43.895380 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.895351    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ecc7b580-81a5-404f-a8f9-3cd655b40a23-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.895557 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.895390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecc7b580-81a5-404f-a8f9-3cd655b40a23-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.895557 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.895428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecc7b580-81a5-404f-a8f9-3cd655b40a23-tmp\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.895557 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.895446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ecc7b580-81a5-404f-a8f9-3cd655b40a23-manifests\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.895557 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.895468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7cr2\" (UniqueName: \"kubernetes.io/projected/ecc7b580-81a5-404f-a8f9-3cd655b40a23-kube-api-access-k7cr2\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.895905 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.895875    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ecc7b580-81a5-404f-a8f9-3cd655b40a23-manifests\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.897854 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.897823    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecc7b580-81a5-404f-a8f9-3cd655b40a23-tmp\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.897854 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.897841    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecc7b580-81a5-404f-a8f9-3cd655b40a23-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.897974 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.897957    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ecc7b580-81a5-404f-a8f9-3cd655b40a23-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.911009 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.910987    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7cr2\" (UniqueName: \"kubernetes.io/projected/ecc7b580-81a5-404f-a8f9-3cd655b40a23-kube-api-access-k7cr2\") pod \"opendatahub-operator-controller-manager-5bf8845876-nclmg\" (UID: \"ecc7b580-81a5-404f-a8f9-3cd655b40a23\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:43.998973 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:43.998942    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:44.052235 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:44.052206    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:44.160197 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:44.160112    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:44.275247 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:44.275211    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:44.333968 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:44.333945    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"]
Sep 11 21:56:44.335385 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:56:44.335360    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc7b580_81a5_404f_a8f9_3cd655b40a23.slice/crio-5b1605a5a0eb9866922a51d386d215b5e8eeb071caf1ad6ed17e5d639e3683cf WatchSource:0}: Error finding container 5b1605a5a0eb9866922a51d386d215b5e8eeb071caf1ad6ed17e5d639e3683cf: Status 404 returned error can't find the container with id 5b1605a5a0eb9866922a51d386d215b5e8eeb071caf1ad6ed17e5d639e3683cf
Sep 11 21:56:44.372190 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:44.372159    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:44.471003 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:44.470923    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:44.571545 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:44.571502    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:45.209647 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:45.209594    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg" event={"ID":"ecc7b580-81a5-404f-a8f9-3cd655b40a23","Type":"ContainerStarted","Data":"5b1605a5a0eb9866922a51d386d215b5e8eeb071caf1ad6ed17e5d639e3683cf"}
Sep 11 21:56:48.225691 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:48.225655    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg" event={"ID":"ecc7b580-81a5-404f-a8f9-3cd655b40a23","Type":"ContainerStarted","Data":"f0721019ae6d36a01731f2339946b766b30c74fd59cd088e8e55edcd54dde721"}
Sep 11 21:56:49.232040 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:49.232002    2574 generic.go:356] "Generic (PLEG): container finished" podID="ecc7b580-81a5-404f-a8f9-3cd655b40a23" containerID="f0721019ae6d36a01731f2339946b766b30c74fd59cd088e8e55edcd54dde721" exitCode=0
Sep 11 21:56:49.232546 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:49.232046    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg" event={"ID":"ecc7b580-81a5-404f-a8f9-3cd655b40a23","Type":"ContainerDied","Data":"f0721019ae6d36a01731f2339946b766b30c74fd59cd088e8e55edcd54dde721"}
Sep 11 21:56:50.238654 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:50.238615    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg" event={"ID":"ecc7b580-81a5-404f-a8f9-3cd655b40a23","Type":"ContainerStarted","Data":"7629a7845e438106411738a10ff066671e145d07b817cccfad2578033471416f"}
Sep 11 21:56:50.239031 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:50.238741    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:56:50.262716 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:50.262665    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg" podStartSLOduration=2.921343777 podStartE2EDuration="7.262650517s" totalImagesPullingTime="3.23193247s" totalInitContainerRuntime="1.10937427s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:56:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:56:50.261757586 +0000 UTC m=+276.996868934" watchObservedRunningTime="2026-09-11 21:56:50.262650517 +0000 UTC m=+276.997761862"
Sep 11 21:56:51.033651 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:51.033622    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:51.133229 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:51.133185    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:58.642304 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:58.642262    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:58.742134 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:58.742099    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:59.079845 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:59.079758    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:59.152571 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:56:59.152524    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:01.245081 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:01.245048    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-nclmg"
Sep 11 21:57:04.205655 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:04.205621    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:04.359844 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:04.359810    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:04.747698 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:04.747660    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:05.294151 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:05.294114    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:06.420982 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:06.420947    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:06.526099 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:06.526065    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:07.476930 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:07.476902    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:11.654250 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:11.654211    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:11.753404 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:11.753371    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:12.462509 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:12.462469    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:12.671608 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:12.671572    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:12.929693 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:12.929659    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:13.632612 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:13.632580    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 21:57:13.638403 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:13.638381    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 21:57:13.638760 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:13.638739    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 21:57:13.643162 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:13.643141    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 21:57:13.644505 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:13.644487    2574 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 21:57:15.183118 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.183088    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-89ngx"]
Sep 11 21:57:15.186287 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.186272    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-89ngx"
Sep 11 21:57:15.189122 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.189103    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qxlm2\""
Sep 11 21:57:15.189293 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.189281    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 11 21:57:15.196204 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.196183    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-89ngx"]
Sep 11 21:57:15.207281 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.207249    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:15.269819 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.269792    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:15.349475 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.349440    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzl44\" (UniqueName: \"kubernetes.io/projected/36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2-kube-api-access-vzl44\") pod \"odh-model-controller-7998c67dc4-89ngx\" (UID: \"36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2\") " pod="opendatahub/odh-model-controller-7998c67dc4-89ngx"
Sep 11 21:57:15.349475 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.349476    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2-cert\") pod \"odh-model-controller-7998c67dc4-89ngx\" (UID: \"36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2\") " pod="opendatahub/odh-model-controller-7998c67dc4-89ngx"
Sep 11 21:57:15.369917 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.369881    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:15.450652 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.450565    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzl44\" (UniqueName: \"kubernetes.io/projected/36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2-kube-api-access-vzl44\") pod \"odh-model-controller-7998c67dc4-89ngx\" (UID: \"36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2\") " pod="opendatahub/odh-model-controller-7998c67dc4-89ngx"
Sep 11 21:57:15.450652 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.450602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2-cert\") pod \"odh-model-controller-7998c67dc4-89ngx\" (UID: \"36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2\") " pod="opendatahub/odh-model-controller-7998c67dc4-89ngx"
Sep 11 21:57:15.450833 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:57:15.450727    2574 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 11 21:57:15.450833 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:57:15.450787    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2-cert podName:36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2 nodeName:}" failed. No retries permitted until 2026-09-11 21:57:15.950771569 +0000 UTC m=+302.685882893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2-cert") pod "odh-model-controller-7998c67dc4-89ngx" (UID: "36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2") : secret "odh-model-controller-webhook-cert" not found
Sep 11 21:57:15.460872 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.460848    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzl44\" (UniqueName: \"kubernetes.io/projected/36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2-kube-api-access-vzl44\") pod \"odh-model-controller-7998c67dc4-89ngx\" (UID: \"36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2\") " pod="opendatahub/odh-model-controller-7998c67dc4-89ngx"
Sep 11 21:57:15.676897 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.676849    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:15.704978 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.704906    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x99wl"]
Sep 11 21:57:15.708290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.708275    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-x99wl"
Sep 11 21:57:15.710969 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.710954    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 11 21:57:15.721066 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.721042    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x99wl"]
Sep 11 21:57:15.855006 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.854974    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crdnb\" (UniqueName: \"kubernetes.io/projected/bd4bad2e-70df-4859-acb3-ea92c3183b12-kube-api-access-crdnb\") pod \"model-serving-api-54d89bd5fc-x99wl\" (UID: \"bd4bad2e-70df-4859-acb3-ea92c3183b12\") " pod="opendatahub/model-serving-api-54d89bd5fc-x99wl"
Sep 11 21:57:15.855184 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.855044    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4bad2e-70df-4859-acb3-ea92c3183b12-tls-certs\") pod \"model-serving-api-54d89bd5fc-x99wl\" (UID: \"bd4bad2e-70df-4859-acb3-ea92c3183b12\") " pod="opendatahub/model-serving-api-54d89bd5fc-x99wl"
Sep 11 21:57:15.956008 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.955927    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crdnb\" (UniqueName: \"kubernetes.io/projected/bd4bad2e-70df-4859-acb3-ea92c3183b12-kube-api-access-crdnb\") pod \"model-serving-api-54d89bd5fc-x99wl\" (UID: \"bd4bad2e-70df-4859-acb3-ea92c3183b12\") " pod="opendatahub/model-serving-api-54d89bd5fc-x99wl"
Sep 11 21:57:15.956008 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.955973    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4bad2e-70df-4859-acb3-ea92c3183b12-tls-certs\") pod \"model-serving-api-54d89bd5fc-x99wl\" (UID: \"bd4bad2e-70df-4859-acb3-ea92c3183b12\") " pod="opendatahub/model-serving-api-54d89bd5fc-x99wl"
Sep 11 21:57:15.956008 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.956006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2-cert\") pod \"odh-model-controller-7998c67dc4-89ngx\" (UID: \"36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2\") " pod="opendatahub/odh-model-controller-7998c67dc4-89ngx"
Sep 11 21:57:15.956215 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:57:15.956136    2574 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 11 21:57:15.956215 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:57:15.956203    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4bad2e-70df-4859-acb3-ea92c3183b12-tls-certs podName:bd4bad2e-70df-4859-acb3-ea92c3183b12 nodeName:}" failed. No retries permitted until 2026-09-11 21:57:16.456186719 +0000 UTC m=+303.191298048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/bd4bad2e-70df-4859-acb3-ea92c3183b12-tls-certs") pod "model-serving-api-54d89bd5fc-x99wl" (UID: "bd4bad2e-70df-4859-acb3-ea92c3183b12") : secret "model-serving-api-tls" not found
Sep 11 21:57:15.958398 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.958382    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2-cert\") pod \"odh-model-controller-7998c67dc4-89ngx\" (UID: \"36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2\") " pod="opendatahub/odh-model-controller-7998c67dc4-89ngx"
Sep 11 21:57:15.966197 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:15.966176    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crdnb\" (UniqueName: \"kubernetes.io/projected/bd4bad2e-70df-4859-acb3-ea92c3183b12-kube-api-access-crdnb\") pod \"model-serving-api-54d89bd5fc-x99wl\" (UID: \"bd4bad2e-70df-4859-acb3-ea92c3183b12\") " pod="opendatahub/model-serving-api-54d89bd5fc-x99wl"
Sep 11 21:57:16.096605 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:16.096564    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-89ngx"
Sep 11 21:57:16.368060 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:16.368034    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-89ngx"]
Sep 11 21:57:16.370528 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:57:16.370502    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f4d8f5_8315_4cbe_9ec2_30a21b0da9c2.slice/crio-f1383faf8365f7005c9c67267ecf9b35fd45148b52c2afd698c71141b91b4abd WatchSource:0}: Error finding container f1383faf8365f7005c9c67267ecf9b35fd45148b52c2afd698c71141b91b4abd: Status 404 returned error can't find the container with id f1383faf8365f7005c9c67267ecf9b35fd45148b52c2afd698c71141b91b4abd
Sep 11 21:57:16.371841 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:16.371824    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:57:16.460644 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:16.460607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4bad2e-70df-4859-acb3-ea92c3183b12-tls-certs\") pod \"model-serving-api-54d89bd5fc-x99wl\" (UID: \"bd4bad2e-70df-4859-acb3-ea92c3183b12\") " pod="opendatahub/model-serving-api-54d89bd5fc-x99wl"
Sep 11 21:57:16.462949 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:16.462931    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4bad2e-70df-4859-acb3-ea92c3183b12-tls-certs\") pod \"model-serving-api-54d89bd5fc-x99wl\" (UID: \"bd4bad2e-70df-4859-acb3-ea92c3183b12\") " pod="opendatahub/model-serving-api-54d89bd5fc-x99wl"
Sep 11 21:57:16.620013 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:16.619920    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-x99wl"
Sep 11 21:57:16.891047 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:16.890919    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x99wl"]
Sep 11 21:57:16.894501 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:57:16.894471    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4bad2e_70df_4859_acb3_ea92c3183b12.slice/crio-20550a8050621fc1f92db16730690bfcb2a4ce3065b651ed82b668fad97746b1 WatchSource:0}: Error finding container 20550a8050621fc1f92db16730690bfcb2a4ce3065b651ed82b668fad97746b1: Status 404 returned error can't find the container with id 20550a8050621fc1f92db16730690bfcb2a4ce3065b651ed82b668fad97746b1
Sep 11 21:57:17.086565 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:17.086533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:17.378178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:17.372045    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-x99wl" event={"ID":"bd4bad2e-70df-4859-acb3-ea92c3183b12","Type":"ContainerStarted","Data":"20550a8050621fc1f92db16730690bfcb2a4ce3065b651ed82b668fad97746b1"}
Sep 11 21:57:17.378178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:17.373967    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-89ngx" event={"ID":"36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2","Type":"ContainerStarted","Data":"f1383faf8365f7005c9c67267ecf9b35fd45148b52c2afd698c71141b91b4abd"}
Sep 11 21:57:17.803939 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:17.803772    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:17.871943 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:17.871908    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:20.389616 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:20.389576    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-x99wl" event={"ID":"bd4bad2e-70df-4859-acb3-ea92c3183b12","Type":"ContainerStarted","Data":"65e718a648c22900e65e7f4a79ba4e32e710f9c18d21a5d536f127a47b86aac6"}
Sep 11 21:57:20.390085 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:20.389709    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-x99wl"
Sep 11 21:57:20.391119 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:20.391089    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-89ngx" event={"ID":"36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2","Type":"ContainerStarted","Data":"d33f935760595420b0335f6ef4e7f80c389574e2a0784cf5bfedfec02e9d8f5f"}
Sep 11 21:57:20.391261 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:20.391215    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-89ngx"
Sep 11 21:57:20.408692 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:20.408650    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-x99wl" podStartSLOduration=2.044420123 podStartE2EDuration="5.408638597s" totalImagesPullingTime="3.364218474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:57:20.405849754 +0000 UTC m=+307.140961110" watchObservedRunningTime="2026-09-11 21:57:20.408638597 +0000 UTC m=+307.143749942"
Sep 11 21:57:20.424575 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:20.424531    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-89ngx" podStartSLOduration=1.577762908 podStartE2EDuration="5.424516615s" totalImagesPullingTime="3.846753707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:57:20.422656223 +0000 UTC m=+307.157767569" watchObservedRunningTime="2026-09-11 21:57:20.424516615 +0000 UTC m=+307.159627960"
Sep 11 21:57:21.970439 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:21.970411    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"]
Sep 11 21:57:21.973864 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:21.973848    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:21.976902 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:21.976876    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 21:57:21.978365 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:21.978348    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 21:57:21.978513 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:21.978479    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 21:57:21.978592 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:21.978507    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 21:57:21.978656 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:21.978623    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-z26f5\""
Sep 11 21:57:21.978656 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:21.978637    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:57:21.984010 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:21.983990    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"]
Sep 11 21:57:22.052210 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.052179    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"]
Sep 11 21:57:22.055498 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.055478    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.061219 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.061184    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xmxph\""
Sep 11 21:57:22.062455 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.062072    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 21:57:22.062455 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.062094    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 21:57:22.062455 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.062130    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 21:57:22.062580 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.062559    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 21:57:22.071569 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.071546    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"]
Sep 11 21:57:22.108238 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.108218    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a490e2da-66cd-43b4-9c38-78f639b34dd1-cert\") pod \"lws-controller-manager-57494c65d5-wfxq5\" (UID: \"a490e2da-66cd-43b4-9c38-78f639b34dd1\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:22.108374 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.108250    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9vh\" (UniqueName: \"kubernetes.io/projected/a490e2da-66cd-43b4-9c38-78f639b34dd1-kube-api-access-vf9vh\") pod \"lws-controller-manager-57494c65d5-wfxq5\" (UID: \"a490e2da-66cd-43b4-9c38-78f639b34dd1\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:22.108374 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.108353    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a490e2da-66cd-43b4-9c38-78f639b34dd1-metrics-cert\") pod \"lws-controller-manager-57494c65d5-wfxq5\" (UID: \"a490e2da-66cd-43b4-9c38-78f639b34dd1\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:22.108374 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.108373    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a490e2da-66cd-43b4-9c38-78f639b34dd1-manager-config\") pod \"lws-controller-manager-57494c65d5-wfxq5\" (UID: \"a490e2da-66cd-43b4-9c38-78f639b34dd1\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:22.208826 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.208794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.208943 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.208834    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a490e2da-66cd-43b4-9c38-78f639b34dd1-cert\") pod \"lws-controller-manager-57494c65d5-wfxq5\" (UID: \"a490e2da-66cd-43b4-9c38-78f639b34dd1\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:22.208943 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.208855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9vh\" (UniqueName: \"kubernetes.io/projected/a490e2da-66cd-43b4-9c38-78f639b34dd1-kube-api-access-vf9vh\") pod \"lws-controller-manager-57494c65d5-wfxq5\" (UID: \"a490e2da-66cd-43b4-9c38-78f639b34dd1\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:22.208943 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.208870    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.208943 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.208912    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.208943 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.208932    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.209210 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.209045    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a490e2da-66cd-43b4-9c38-78f639b34dd1-metrics-cert\") pod \"lws-controller-manager-57494c65d5-wfxq5\" (UID: \"a490e2da-66cd-43b4-9c38-78f639b34dd1\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:22.209210 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.209086    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a490e2da-66cd-43b4-9c38-78f639b34dd1-manager-config\") pod \"lws-controller-manager-57494c65d5-wfxq5\" (UID: \"a490e2da-66cd-43b4-9c38-78f639b34dd1\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:22.209210 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.209118    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.209210 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.209199    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.209401 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.209235    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frkg7\" (UniqueName: \"kubernetes.io/projected/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-kube-api-access-frkg7\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.209707 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.209688    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a490e2da-66cd-43b4-9c38-78f639b34dd1-manager-config\") pod \"lws-controller-manager-57494c65d5-wfxq5\" (UID: \"a490e2da-66cd-43b4-9c38-78f639b34dd1\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:22.211298 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.211278    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a490e2da-66cd-43b4-9c38-78f639b34dd1-metrics-cert\") pod \"lws-controller-manager-57494c65d5-wfxq5\" (UID: \"a490e2da-66cd-43b4-9c38-78f639b34dd1\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:22.211411 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.211322    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a490e2da-66cd-43b4-9c38-78f639b34dd1-cert\") pod \"lws-controller-manager-57494c65d5-wfxq5\" (UID: \"a490e2da-66cd-43b4-9c38-78f639b34dd1\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:22.224011 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.223958    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9vh\" (UniqueName: \"kubernetes.io/projected/a490e2da-66cd-43b4-9c38-78f639b34dd1-kube-api-access-vf9vh\") pod \"lws-controller-manager-57494c65d5-wfxq5\" (UID: \"a490e2da-66cd-43b4-9c38-78f639b34dd1\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:22.283410 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.283387    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:22.310090 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.310066    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.310191 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.310107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.310191 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.310130    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkg7\" (UniqueName: \"kubernetes.io/projected/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-kube-api-access-frkg7\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.310191 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.310158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.310191 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.310180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.310439 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.310200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.310439 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.310233    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.310923 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.310894    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.312491 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.312469    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.312652 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.312548    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.312790 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.312766    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.312983 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.312968    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.329903 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.329877    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frkg7\" (UniqueName: \"kubernetes.io/projected/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-kube-api-access-frkg7\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.330372 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.330357    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-96pjh\" (UID: \"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.365485 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.365457    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:22.562861 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.562840    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"]
Sep 11 21:57:22.564941 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:57:22.564906    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda490e2da_66cd_43b4_9c38_78f639b34dd1.slice/crio-a834708091d4f41cce68a8a7e7132798929be0f2c8044a33c325be21a88783f2 WatchSource:0}: Error finding container a834708091d4f41cce68a8a7e7132798929be0f2c8044a33c325be21a88783f2: Status 404 returned error can't find the container with id a834708091d4f41cce68a8a7e7132798929be0f2c8044a33c325be21a88783f2
Sep 11 21:57:22.658624 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:22.658601    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"]
Sep 11 21:57:22.660335 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:57:22.660291    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af5c03f_e7b8_45fc_9f9b_cbf90511d3c7.slice/crio-6d47d999f7651bfc9e981cb8e7b29abc521a849d693c37ae3daddff4ea08d527 WatchSource:0}: Error finding container 6d47d999f7651bfc9e981cb8e7b29abc521a849d693c37ae3daddff4ea08d527: Status 404 returned error can't find the container with id 6d47d999f7651bfc9e981cb8e7b29abc521a849d693c37ae3daddff4ea08d527
Sep 11 21:57:23.411015 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:23.410889    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh" event={"ID":"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7","Type":"ContainerStarted","Data":"6d47d999f7651bfc9e981cb8e7b29abc521a849d693c37ae3daddff4ea08d527"}
Sep 11 21:57:23.413611 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:23.413550    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5" event={"ID":"a490e2da-66cd-43b4-9c38-78f639b34dd1","Type":"ContainerStarted","Data":"a834708091d4f41cce68a8a7e7132798929be0f2c8044a33c325be21a88783f2"}
Sep 11 21:57:25.473855 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:25.473818    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:25.639443 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:25.639405    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:25.874608 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:25.874575    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:25.994253 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:25.994215    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:57:25.994357 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:25.994282    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:57:25.996869 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:25.996835    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:26.132210 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:26.132092    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:26.239628 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:26.239596    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:26.431250 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:26.431158    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh" event={"ID":"4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7","Type":"ContainerStarted","Data":"45f25a8f0e62884e99090de3fb4269566204851543112d0166f4a4c1446e602d"}
Sep 11 21:57:26.431437 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:26.431338    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:26.432821 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:26.432795    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5" event={"ID":"a490e2da-66cd-43b4-9c38-78f639b34dd1","Type":"ContainerStarted","Data":"d6da79309675382dd7650ae56fc846685612a8fde787953a24fd9a27f7061334"}
Sep 11 21:57:26.432959 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:26.432860    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:57:26.465531 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:26.465481    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh" podStartSLOduration=1.133702866 podStartE2EDuration="4.465463341s" totalImagesPullingTime="3.331760475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:57:26.46492548 +0000 UTC m=+313.200036851" watchObservedRunningTime="2026-09-11 21:57:26.465463341 +0000 UTC m=+313.200574684"
Sep 11 21:57:26.495190 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:26.495138    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5" podStartSLOduration=2.069432967 podStartE2EDuration="5.495124528s" totalImagesPullingTime="3.425691561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:57:26.493040125 +0000 UTC m=+313.228151473" watchObservedRunningTime="2026-09-11 21:57:26.495124528 +0000 UTC m=+313.230235885"
Sep 11 21:57:27.439191 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:27.439161    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-96pjh"
Sep 11 21:57:31.397679 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:31.397651    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-89ngx"
Sep 11 21:57:31.399667 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:31.399645    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-x99wl"
Sep 11 21:57:32.731872 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:32.731834    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:32.829898 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:32.829839    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:37.439137 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:57:37.439104    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-wfxq5"
Sep 11 21:58:16.295754 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:16.295642    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:16.390229 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:16.390194    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:47.412184 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:47.412141    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:47.470730 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:47.470701    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:52.606262 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:52.606222    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:53.411689 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:53.411653    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:54.209557 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:54.209525    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:57.484909 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:57.484876    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:57.567047 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:57.567011    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:57.675124 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:57.675092    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:58.295168 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:58.295130    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:58.393065 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:58.393034    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:58.492373 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:58.492336    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:59.624708 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:59.624671    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:59.946655 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:58:59.946563    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:01.306847 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:01.306816    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:01.491895 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:01.491862    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:02.032405 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:02.032373    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:03.460812 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:03.460781    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:05.095145 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:05.095112    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:05.575867 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:05.575786    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:07.814509 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:07.814477    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:07.880030 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:07.880005    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:08.210931 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:08.210897    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:08.313099 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:08.313058    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:08.615432 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:08.615394    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:08.715810 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:08.715780    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:15.838237 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:15.838205    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:15.936352 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:15.936297    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:16.331121 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.331045    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7pf2"]
Sep 11 21:59:16.334152 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.334136    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7pf2"
Sep 11 21:59:16.337385 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.337363    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 21:59:16.337515 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.337464    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8bxkr\""
Sep 11 21:59:16.337515 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.337478    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 21:59:16.348038 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.348017    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7pf2"]
Sep 11 21:59:16.348235 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:59:16.348217    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-tsjlm], unattached volumes=[], failed to process volumes=[extensions-socket-volume kube-api-access-tsjlm]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7pf2" podUID="c369410d-95f6-4432-8162-7404fea3bd00"
Sep 11 21:59:16.360433 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.360413    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7pf2"]
Sep 11 21:59:16.404467 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.404444    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"]
Sep 11 21:59:16.407499 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.407484    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"
Sep 11 21:59:16.501009 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.500980    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"]
Sep 11 21:59:16.530004 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.529964    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/385caa9e-eaad-4bdb-8a5d-d5697d286cce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-b6lgc\" (UID: \"385caa9e-eaad-4bdb-8a5d-d5697d286cce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"
Sep 11 21:59:16.530178 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.530011    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpkrf\" (UniqueName: \"kubernetes.io/projected/385caa9e-eaad-4bdb-8a5d-d5697d286cce-kube-api-access-hpkrf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-b6lgc\" (UID: \"385caa9e-eaad-4bdb-8a5d-d5697d286cce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"
Sep 11 21:59:16.631112 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.631081    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpkrf\" (UniqueName: \"kubernetes.io/projected/385caa9e-eaad-4bdb-8a5d-d5697d286cce-kube-api-access-hpkrf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-b6lgc\" (UID: \"385caa9e-eaad-4bdb-8a5d-d5697d286cce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"
Sep 11 21:59:16.631267 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.631156    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/385caa9e-eaad-4bdb-8a5d-d5697d286cce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-b6lgc\" (UID: \"385caa9e-eaad-4bdb-8a5d-d5697d286cce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"
Sep 11 21:59:16.631486 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.631469    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/385caa9e-eaad-4bdb-8a5d-d5697d286cce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-b6lgc\" (UID: \"385caa9e-eaad-4bdb-8a5d-d5697d286cce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"
Sep 11 21:59:16.644495 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.644467    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpkrf\" (UniqueName: \"kubernetes.io/projected/385caa9e-eaad-4bdb-8a5d-d5697d286cce-kube-api-access-hpkrf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-b6lgc\" (UID: \"385caa9e-eaad-4bdb-8a5d-d5697d286cce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"
Sep 11 21:59:16.717513 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.717485    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"
Sep 11 21:59:16.925548 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.925469    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7pf2"
Sep 11 21:59:16.928381 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:59:16.928353    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-d7pf2\" is forbidden: User \"system:node:ip-10-0-128-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-201.ec2.internal' and this object" podUID="c369410d-95f6-4432-8162-7404fea3bd00" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7pf2"
Sep 11 21:59:16.932602 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.932583    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7pf2"
Sep 11 21:59:16.935273 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:59:16.935252    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-d7pf2\" is forbidden: User \"system:node:ip-10-0-128-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-201.ec2.internal' and this object" podUID="c369410d-95f6-4432-8162-7404fea3bd00" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7pf2"
Sep 11 21:59:16.994079 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:16.994048    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"]
Sep 11 21:59:16.998678 ip-10-0-128-201 kubenswrapper[2574]: W0911 21:59:16.998652    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385caa9e_eaad_4bdb_8a5d_d5697d286cce.slice/crio-ff42f4ba2ab5e9654f55c14e22141e26c3d94143eb93ecbdda414348ad8eae8d WatchSource:0}: Error finding container ff42f4ba2ab5e9654f55c14e22141e26c3d94143eb93ecbdda414348ad8eae8d: Status 404 returned error can't find the container with id ff42f4ba2ab5e9654f55c14e22141e26c3d94143eb93ecbdda414348ad8eae8d
Sep 11 21:59:17.720359 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:17.720302    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c369410d-95f6-4432-8162-7404fea3bd00" path="/var/lib/kubelet/pods/c369410d-95f6-4432-8162-7404fea3bd00/volumes"
Sep 11 21:59:17.934818 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:17.934776    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc" event={"ID":"385caa9e-eaad-4bdb-8a5d-d5697d286cce","Type":"ContainerStarted","Data":"ff42f4ba2ab5e9654f55c14e22141e26c3d94143eb93ecbdda414348ad8eae8d"}
Sep 11 21:59:17.935245 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:17.934832    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7pf2"
Sep 11 21:59:17.940351 ip-10-0-128-201 kubenswrapper[2574]: E0911 21:59:17.940305    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-d7pf2\" is forbidden: User \"system:node:ip-10-0-128-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-201.ec2.internal' and this object" podUID="c369410d-95f6-4432-8162-7404fea3bd00" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7pf2"
Sep 11 21:59:20.038018 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:20.037960    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:20.135954 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:20.135918    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:20.807939 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:20.807901    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:20.910189 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:20.910140    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:21.963125 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:21.963090    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc" event={"ID":"385caa9e-eaad-4bdb-8a5d-d5697d286cce","Type":"ContainerStarted","Data":"dcd890be7164cf551b4a1904171a56a09c55e156fee1217798c4301d998ececf"}
Sep 11 21:59:21.963503 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:21.963179    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"
Sep 11 21:59:21.988210 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:21.988165    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc" podStartSLOduration=1.599427025 podStartE2EDuration="5.98815148s" totalImagesPullingTime="4.388724455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:59:21.985067441 +0000 UTC m=+428.720178789" watchObservedRunningTime="2026-09-11 21:59:21.98815148 +0000 UTC m=+428.723262838"
Sep 11 21:59:24.808823 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:24.808794    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:24.929289 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:24.929255    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:25.403995 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:25.403957    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:25.468242 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:25.468210    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:25.846265 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:25.846228    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:31.843587 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:31.843551    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:31.908376 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:31.908337    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:32.218536 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:32.218427    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:32.279834 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:32.279793    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:32.614782 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:32.614750    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:32.694239 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:32.694195    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:32.970154 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:32.970070    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"
Sep 11 21:59:38.086098 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:38.086009    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:38.142429 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:38.142397    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:42.839811 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:42.839773    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:42.940651 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:42.940610    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:47.651290 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:47.651255    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:47.747742 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:47.747700    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:49.456350 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:49.456291    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:50.228410 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:50.228379    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:59.435421 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:59.435386    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:59.533274 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:59.533213    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:59.635579 ip-10-0-128-201 kubenswrapper[2574]: I0911 21:59:59.635541    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:00.367932 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:00.367898    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:00.451047 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:00.451014    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:15.639363 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:15.639323    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:15.736669 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:15.736635    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:16.194809 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:16.194776    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:16.536784 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:16.536685    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:16.637037 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:16.636998    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:23.557683 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:23.557641    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:23.657514 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:23.657483    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:30.635158 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:30.635107    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:30.807407 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:30.807370    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:39.473349 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:39.473290    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:39.588828 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:39.588791    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:39.691668 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:39.691632    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:39.754627 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:39.754533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:41.027783 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:41.027753    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:43.014436 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:43.014401    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:43.115146 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:00:43.115114    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:14.551070 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:14.550988    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:14.665290 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:14.665258    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:16.113728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.113692    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:16.214946 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.214915    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:16.288523 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.288493    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:16.616592 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.616563    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:16.718720 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.718685    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:16.719457 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.719438    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp"]
Sep 11 22:01:16.721799 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.721785    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp"
Sep 11 22:01:16.724798 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.724782    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-w5hcf\""
Sep 11 22:01:16.725026 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.725012    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 22:01:16.734443 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.734420    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp"]
Sep 11 22:01:16.812943 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.812911    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:16.816439 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.816399    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78a736e-b63b-48f0-a7ee-b378f2723cb4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-k5dpp\" (UID: \"a78a736e-b63b-48f0-a7ee-b378f2723cb4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp"
Sep 11 22:01:16.816627 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.816524    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnv2k\" (UniqueName: \"kubernetes.io/projected/a78a736e-b63b-48f0-a7ee-b378f2723cb4-kube-api-access-xnv2k\") pod \"payload-pre-processing-59fbc47bf8-k5dpp\" (UID: \"a78a736e-b63b-48f0-a7ee-b378f2723cb4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp"
Sep 11 22:01:16.917381 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.917273    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78a736e-b63b-48f0-a7ee-b378f2723cb4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-k5dpp\" (UID: \"a78a736e-b63b-48f0-a7ee-b378f2723cb4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp"
Sep 11 22:01:16.917381 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.917355    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnv2k\" (UniqueName: \"kubernetes.io/projected/a78a736e-b63b-48f0-a7ee-b378f2723cb4-kube-api-access-xnv2k\") pod \"payload-pre-processing-59fbc47bf8-k5dpp\" (UID: \"a78a736e-b63b-48f0-a7ee-b378f2723cb4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp"
Sep 11 22:01:16.918017 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.917990    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78a736e-b63b-48f0-a7ee-b378f2723cb4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-k5dpp\" (UID: \"a78a736e-b63b-48f0-a7ee-b378f2723cb4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp"
Sep 11 22:01:16.930075 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:16.930044    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnv2k\" (UniqueName: \"kubernetes.io/projected/a78a736e-b63b-48f0-a7ee-b378f2723cb4-kube-api-access-xnv2k\") pod \"payload-pre-processing-59fbc47bf8-k5dpp\" (UID: \"a78a736e-b63b-48f0-a7ee-b378f2723cb4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp"
Sep 11 22:01:17.032320 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:17.032281    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp"
Sep 11 22:01:17.300199 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:17.300149    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp"]
Sep 11 22:01:17.302443 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:01:17.302416    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78a736e_b63b_48f0_a7ee_b378f2723cb4.slice/crio-3d31b7c9faa9bf1105bbf304a6cc633fd2721f47c0868216b03d1654b2da887f WatchSource:0}: Error finding container 3d31b7c9faa9bf1105bbf304a6cc633fd2721f47c0868216b03d1654b2da887f: Status 404 returned error can't find the container with id 3d31b7c9faa9bf1105bbf304a6cc633fd2721f47c0868216b03d1654b2da887f
Sep 11 22:01:17.496631 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:17.496596    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp" event={"ID":"a78a736e-b63b-48f0-a7ee-b378f2723cb4","Type":"ContainerStarted","Data":"3d31b7c9faa9bf1105bbf304a6cc633fd2721f47c0868216b03d1654b2da887f"}
Sep 11 22:01:20.512466 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:20.512432    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp" event={"ID":"a78a736e-b63b-48f0-a7ee-b378f2723cb4","Type":"ContainerStarted","Data":"4e162a9693b7da5bb295f9ccd85abf6e9d76002611dc07b46eb50594028c9ece"}
Sep 11 22:01:20.512863 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:20.512554    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp"
Sep 11 22:01:20.531059 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:20.531011    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp" podStartSLOduration=2.300053991 podStartE2EDuration="4.530998145s" totalImagesPullingTime="2.230944154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:20.529610188 +0000 UTC m=+547.264721535" watchObservedRunningTime="2026-09-11 22:01:20.530998145 +0000 UTC m=+547.266109535"
Sep 11 22:01:26.880120 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:26.880085    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:26.957402 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:26.957362    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:30.130483 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:30.130438    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:31.518776 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:31.518745    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k5dpp"
Sep 11 22:01:32.410712 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:32.410671    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:37.890394 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:37.890356    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:45.966124 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:45.966084    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"]
Sep 11 22:01:45.972061 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:45.972037    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:45.976666 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:45.976641    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 22:01:45.977070 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:45.977050    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 22:01:45.977645 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:45.977620    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l4wd6\""
Sep 11 22:01:45.977856 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:45.977839    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 22:01:45.978815 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:45.978777    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"]
Sep 11 22:01:46.008542 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.008510    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:46.064889 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.064862    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5790506-7835-4c5e-af3a-f5a4425d5814-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.065044 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.064893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5790506-7835-4c5e-af3a-f5a4425d5814-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.065044 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.064924    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5790506-7835-4c5e-af3a-f5a4425d5814-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.065044 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.064941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s49hd\" (UniqueName: \"kubernetes.io/projected/c5790506-7835-4c5e-af3a-f5a4425d5814-kube-api-access-s49hd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.065044 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.065013    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5790506-7835-4c5e-af3a-f5a4425d5814-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.065195 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.065045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5790506-7835-4c5e-af3a-f5a4425d5814-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.166400 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.166363    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5790506-7835-4c5e-af3a-f5a4425d5814-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.166400 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.166403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5790506-7835-4c5e-af3a-f5a4425d5814-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.166585 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.166476    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5790506-7835-4c5e-af3a-f5a4425d5814-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.166585 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.166511    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5790506-7835-4c5e-af3a-f5a4425d5814-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.166585 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.166558    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5790506-7835-4c5e-af3a-f5a4425d5814-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.166698 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.166585    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s49hd\" (UniqueName: \"kubernetes.io/projected/c5790506-7835-4c5e-af3a-f5a4425d5814-kube-api-access-s49hd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.166851 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.166830    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5790506-7835-4c5e-af3a-f5a4425d5814-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.166893 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.166866    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5790506-7835-4c5e-af3a-f5a4425d5814-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.166928 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.166915    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5790506-7835-4c5e-af3a-f5a4425d5814-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.168741 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.168709    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5790506-7835-4c5e-af3a-f5a4425d5814-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.168993 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.168973    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5790506-7835-4c5e-af3a-f5a4425d5814-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.175192 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.175173    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s49hd\" (UniqueName: \"kubernetes.io/projected/c5790506-7835-4c5e-af3a-f5a4425d5814-kube-api-access-s49hd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jw44h\" (UID: \"c5790506-7835-4c5e-af3a-f5a4425d5814\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.284555 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.284477    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:46.558906 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.558884    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"]
Sep 11 22:01:46.561030 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:01:46.561005    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5790506_7835_4c5e_af3a_f5a4425d5814.slice/crio-aa22388e91caeb2fc7f424c3af02c55a44bba9400beca29664c1f0cd8dc33449 WatchSource:0}: Error finding container aa22388e91caeb2fc7f424c3af02c55a44bba9400beca29664c1f0cd8dc33449: Status 404 returned error can't find the container with id aa22388e91caeb2fc7f424c3af02c55a44bba9400beca29664c1f0cd8dc33449
Sep 11 22:01:46.634755 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:46.634724    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h" event={"ID":"c5790506-7835-4c5e-af3a-f5a4425d5814","Type":"ContainerStarted","Data":"aa22388e91caeb2fc7f424c3af02c55a44bba9400beca29664c1f0cd8dc33449"}
Sep 11 22:01:47.640799 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:47.640764    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h" event={"ID":"c5790506-7835-4c5e-af3a-f5a4425d5814","Type":"ContainerStarted","Data":"c72186781f61effbc0e90325602abd2315c289c59904315dcf32598f82856e06"}
Sep 11 22:01:47.659988 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:47.659933    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h" podStartSLOduration=1.669193377 podStartE2EDuration="2.659912529s" totalImagesPullingTime="990.719152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:47.659598026 +0000 UTC m=+574.394709372" watchObservedRunningTime="2026-09-11 22:01:47.659912529 +0000 UTC m=+574.395023874"
Sep 11 22:01:52.484680 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.484648    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"]
Sep 11 22:01:52.488098 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.488079    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.491018 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.491000    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 22:01:52.506302 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.506273    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"]
Sep 11 22:01:52.517573 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.517538    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:52.518594 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.518572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.518691 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.518605    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.518691 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.518632    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.518819 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.518694    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.518819 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.518709    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.518819 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.518741    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27z7b\" (UniqueName: \"kubernetes.io/projected/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-kube-api-access-27z7b\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.619790 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.619727    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.619790 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.619759    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.619790 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.619775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27z7b\" (UniqueName: \"kubernetes.io/projected/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-kube-api-access-27z7b\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.619991 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.619798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.619991 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.619928    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.620084 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.620002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.620237 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.620215    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.620237 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.620228    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.620396 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.620380    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.621951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.621932    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.622247 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.622233    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.627958 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.627941    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27z7b\" (UniqueName: \"kubernetes.io/projected/7ddc4933-1d12-46e3-a34a-bc6935e9d01d-kube-api-access-27z7b\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs\" (UID: \"7ddc4933-1d12-46e3-a34a-bc6935e9d01d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:52.798610 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:52.798580    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:01:53.071170 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:53.071082    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"]
Sep 11 22:01:53.075141 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:01:53.075106    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddc4933_1d12_46e3_a34a_bc6935e9d01d.slice/crio-39eee799f12866e732575da283729f64c3f3883da6b2267219e662223ea60942 WatchSource:0}: Error finding container 39eee799f12866e732575da283729f64c3f3883da6b2267219e662223ea60942: Status 404 returned error can't find the container with id 39eee799f12866e732575da283729f64c3f3883da6b2267219e662223ea60942
Sep 11 22:01:53.172322 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:53.172280    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:53.670000 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:53.669961    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs" event={"ID":"7ddc4933-1d12-46e3-a34a-bc6935e9d01d","Type":"ContainerStarted","Data":"4c38d0af659bc6e07764dbe3fc4b884fb37dca13bb9314897153861436019214"}
Sep 11 22:01:53.670370 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:53.670019    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs" event={"ID":"7ddc4933-1d12-46e3-a34a-bc6935e9d01d","Type":"ContainerStarted","Data":"39eee799f12866e732575da283729f64c3f3883da6b2267219e662223ea60942"}
Sep 11 22:01:53.687840 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:53.687791    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs" podStartSLOduration=1.490145855 podStartE2EDuration="1.687778097s" totalImagesPullingTime="197.632242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:53.686239732 +0000 UTC m=+580.421351079" watchObservedRunningTime="2026-09-11 22:01:53.687778097 +0000 UTC m=+580.422889442"
Sep 11 22:01:56.285453 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:56.285407    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:56.285838 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:56.285488    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:56.298028 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:56.298005    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:56.695134 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:56.695107    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jw44h"
Sep 11 22:01:58.595778 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:01:58.595736    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:02.799604 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:02.799572    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:02:02.799996 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:02.799623    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:02:02.811838 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:02.811813    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:02:03.728902 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:03.728876    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs"
Sep 11 22:02:04.836483 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:04.836435    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:12.617557 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:12.617513    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:13.667656 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:13.667622    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:02:13.668097 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:13.668081    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:02:13.671728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:13.671706    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:02:13.675130 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:13.675110    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:02:15.970804 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:15.970769    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"]
Sep 11 22:02:15.974143 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:15.974115    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:15.976749 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:15.976726    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 22:02:15.983872 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:15.983848    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"]
Sep 11 22:02:16.086419 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.086387    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:16.124389 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.124362    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f771b808-1542-49f5-b934-6d0ec3142608-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.124513 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.124396    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f771b808-1542-49f5-b934-6d0ec3142608-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.124513 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.124433    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f771b808-1542-49f5-b934-6d0ec3142608-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.124513 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.124486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f771b808-1542-49f5-b934-6d0ec3142608-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.124618 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.124548    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f771b808-1542-49f5-b934-6d0ec3142608-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.124618 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.124594    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6trlg\" (UniqueName: \"kubernetes.io/projected/f771b808-1542-49f5-b934-6d0ec3142608-kube-api-access-6trlg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.225601 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.225530    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f771b808-1542-49f5-b934-6d0ec3142608-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.225601 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.225580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6trlg\" (UniqueName: \"kubernetes.io/projected/f771b808-1542-49f5-b934-6d0ec3142608-kube-api-access-6trlg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.225804 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.225613    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f771b808-1542-49f5-b934-6d0ec3142608-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.225804 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.225633    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f771b808-1542-49f5-b934-6d0ec3142608-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.225804 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.225656    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f771b808-1542-49f5-b934-6d0ec3142608-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.225804 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.225673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f771b808-1542-49f5-b934-6d0ec3142608-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.226160 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.226134    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f771b808-1542-49f5-b934-6d0ec3142608-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.226321 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.226179    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f771b808-1542-49f5-b934-6d0ec3142608-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.226321 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.226188    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f771b808-1542-49f5-b934-6d0ec3142608-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.227920 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.227895    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f771b808-1542-49f5-b934-6d0ec3142608-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.228112 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.228096    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f771b808-1542-49f5-b934-6d0ec3142608-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.233552 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.233525    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6trlg\" (UniqueName: \"kubernetes.io/projected/f771b808-1542-49f5-b934-6d0ec3142608-kube-api-access-6trlg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6qwn\" (UID: \"f771b808-1542-49f5-b934-6d0ec3142608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.285021 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.284998    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:16.557410 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.557337    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"]
Sep 11 22:02:16.560540 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:02:16.560516    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf771b808_1542_49f5_b934_6d0ec3142608.slice/crio-49524b0939238d3f1eb25f10b02c2a5f32ef6daecc3dd0ee8d1708dce30d366d WatchSource:0}: Error finding container 49524b0939238d3f1eb25f10b02c2a5f32ef6daecc3dd0ee8d1708dce30d366d: Status 404 returned error can't find the container with id 49524b0939238d3f1eb25f10b02c2a5f32ef6daecc3dd0ee8d1708dce30d366d
Sep 11 22:02:16.562357 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.562336    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:02:16.777983 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:16.777960    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn" event={"ID":"f771b808-1542-49f5-b934-6d0ec3142608","Type":"ContainerStarted","Data":"49524b0939238d3f1eb25f10b02c2a5f32ef6daecc3dd0ee8d1708dce30d366d"}
Sep 11 22:02:17.783654 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:17.783615    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn" event={"ID":"f771b808-1542-49f5-b934-6d0ec3142608","Type":"ContainerStarted","Data":"acb88fcf13deabb5941b4f33b50a82113deb194f1a7b20489c19da4e92aee14d"}
Sep 11 22:02:17.800525 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:17.800474    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn" podStartSLOduration=2.631532285 podStartE2EDuration="2.800458812s" totalImagesPullingTime="168.926527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:02:17.799248141 +0000 UTC m=+604.534359490" watchObservedRunningTime="2026-09-11 22:02:17.800458812 +0000 UTC m=+604.535570157"
Sep 11 22:02:26.285640 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:26.285607    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:26.286086 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:26.285693    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:26.297641 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:26.297613    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:26.835776 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:26.835748    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6qwn"
Sep 11 22:02:47.420625 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:02:47.420592    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:06.140568 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:06.140537    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:06.203340 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:06.203290    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:40.123114 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:40.123079    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:40.213567 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:40.213537    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:41.303666 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:41.303627    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:41.399068 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:41.399032    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:52.996993 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:52.996955    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:56.675548 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:56.675509    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:56.776518 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:56.776485    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:56.878151 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:56.878107    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:57.036760 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:57.036675    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:57.147777 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:57.147743    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:57.234277 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:57.234236    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:57.364186 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:57.364156    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:57.465563 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:57.465528    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:57.731277 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:57.731190    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:58.132943 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:58.132908    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:58.228100 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:58.228071    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:58.588245 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:03:58.588208    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:00.397957 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:00.397924    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:00.494361 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:00.494331    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:00.635852 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:00.635821    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:00.735167 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:00.735084    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:00.828330 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:00.828279    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:01.605097 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:01.605059    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:01.704926 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:01.704897    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:02.017608 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.017530    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:02.283671 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.283594    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:02.372080 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.372030    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:02.376937 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.376914    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"]
Sep 11 22:04:02.380267 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.380250    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"
Sep 11 22:04:02.383162 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.383132    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-7e2add\""
Sep 11 22:04:02.391837 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.391813    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"]
Sep 11 22:04:02.512595 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.512555    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"]
Sep 11 22:04:02.513177 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.513150    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de5e2bd-d55f-4ef5-9cf4-58d36c4964af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml\" (UID: \"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"
Sep 11 22:04:02.513318 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.513281    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfrh2\" (UniqueName: \"kubernetes.io/projected/8de5e2bd-d55f-4ef5-9cf4-58d36c4964af-kube-api-access-kfrh2\") pod \"payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml\" (UID: \"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"
Sep 11 22:04:02.521048 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.517375    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"
Sep 11 22:04:02.523748 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.523727    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-7e2add-dockercfg-k88rb\""
Sep 11 22:04:02.528768 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.528746    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"]
Sep 11 22:04:02.614811 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.614778    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfrh2\" (UniqueName: \"kubernetes.io/projected/8de5e2bd-d55f-4ef5-9cf4-58d36c4964af-kube-api-access-kfrh2\") pod \"payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml\" (UID: \"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"
Sep 11 22:04:02.615161 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.614892    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de5e2bd-d55f-4ef5-9cf4-58d36c4964af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml\" (UID: \"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"
Sep 11 22:04:02.615161 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.614921    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjb8\" (UniqueName: \"kubernetes.io/projected/7c9bfb9c-29cd-4b02-805b-35cdd9897df4-kube-api-access-msjb8\") pod \"payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t\" (UID: \"7c9bfb9c-29cd-4b02-805b-35cdd9897df4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"
Sep 11 22:04:02.615161 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.614949    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9bfb9c-29cd-4b02-805b-35cdd9897df4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t\" (UID: \"7c9bfb9c-29cd-4b02-805b-35cdd9897df4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"
Sep 11 22:04:02.615854 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.615835    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de5e2bd-d55f-4ef5-9cf4-58d36c4964af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml\" (UID: \"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"
Sep 11 22:04:02.623252 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.623228    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfrh2\" (UniqueName: \"kubernetes.io/projected/8de5e2bd-d55f-4ef5-9cf4-58d36c4964af-kube-api-access-kfrh2\") pod \"payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml\" (UID: \"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"
Sep 11 22:04:02.691242 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.691216    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"
Sep 11 22:04:02.715929 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.715896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9bfb9c-29cd-4b02-805b-35cdd9897df4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t\" (UID: \"7c9bfb9c-29cd-4b02-805b-35cdd9897df4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"
Sep 11 22:04:02.716075 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.716059    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjb8\" (UniqueName: \"kubernetes.io/projected/7c9bfb9c-29cd-4b02-805b-35cdd9897df4-kube-api-access-msjb8\") pod \"payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t\" (UID: \"7c9bfb9c-29cd-4b02-805b-35cdd9897df4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"
Sep 11 22:04:02.716829 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.716806    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9bfb9c-29cd-4b02-805b-35cdd9897df4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t\" (UID: \"7c9bfb9c-29cd-4b02-805b-35cdd9897df4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"
Sep 11 22:04:02.727769 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.727740    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjb8\" (UniqueName: \"kubernetes.io/projected/7c9bfb9c-29cd-4b02-805b-35cdd9897df4-kube-api-access-msjb8\") pod \"payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t\" (UID: \"7c9bfb9c-29cd-4b02-805b-35cdd9897df4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"
Sep 11 22:04:02.735174 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.735038    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:02.814682 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.814631    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:02.831695 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.831663    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"
Sep 11 22:04:02.937783 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.937749    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:02.979065 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:02.979040    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"]
Sep 11 22:04:02.981660 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:04:02.981632    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de5e2bd_d55f_4ef5_9cf4_58d36c4964af.slice/crio-fc4056913a3f18ad37c75e7c82baa629688d1a980858388ecfc2978f80d9fdba WatchSource:0}: Error finding container fc4056913a3f18ad37c75e7c82baa629688d1a980858388ecfc2978f80d9fdba: Status 404 returned error can't find the container with id fc4056913a3f18ad37c75e7c82baa629688d1a980858388ecfc2978f80d9fdba
Sep 11 22:04:03.101565 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.101540    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"]
Sep 11 22:04:03.103158 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:04:03.103130    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9bfb9c_29cd_4b02_805b_35cdd9897df4.slice/crio-3e67ae4a8cdc4ddc069174f450702a6158c886e9f4b76d7e379292d64dba6925 WatchSource:0}: Error finding container 3e67ae4a8cdc4ddc069174f450702a6158c886e9f4b76d7e379292d64dba6925: Status 404 returned error can't find the container with id 3e67ae4a8cdc4ddc069174f450702a6158c886e9f4b76d7e379292d64dba6925
Sep 11 22:04:03.212284 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.212258    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:03.282347 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.282302    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml" event={"ID":"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af","Type":"ContainerStarted","Data":"5dd9412890d50635fa71df5b3be7b8c90764f2b9f7eca9b22052e347182e5a99"}
Sep 11 22:04:03.282523 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.282379    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"
Sep 11 22:04:03.282523 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.282407    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml" event={"ID":"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af","Type":"ContainerStarted","Data":"fc4056913a3f18ad37c75e7c82baa629688d1a980858388ecfc2978f80d9fdba"}
Sep 11 22:04:03.283963 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.283941    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t" event={"ID":"7c9bfb9c-29cd-4b02-805b-35cdd9897df4","Type":"ContainerStarted","Data":"5c2ba16a9d261fa53b684b0ff60db177ccdf6b4e55bf325cf6f12eaf72ce1efc"}
Sep 11 22:04:03.284074 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.283976    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t" event={"ID":"7c9bfb9c-29cd-4b02-805b-35cdd9897df4","Type":"ContainerStarted","Data":"3e67ae4a8cdc4ddc069174f450702a6158c886e9f4b76d7e379292d64dba6925"}
Sep 11 22:04:03.284146 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.284073    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"
Sep 11 22:04:03.301902 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.301875    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:03.302423 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.302373    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml" podStartSLOduration=1.302358143 podStartE2EDuration="1.302358143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:03.299940836 +0000 UTC m=+710.035052182" watchObservedRunningTime="2026-09-11 22:04:03.302358143 +0000 UTC m=+710.037469491"
Sep 11 22:04:03.317119 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.317077    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t" podStartSLOduration=1.317064111 podStartE2EDuration="1.317064111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:03.315812693 +0000 UTC m=+710.050924039" watchObservedRunningTime="2026-09-11 22:04:03.317064111 +0000 UTC m=+710.052175488"
Sep 11 22:04:03.342936 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.342907    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"]
Sep 11 22:04:03.346878 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.346860    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"
Sep 11 22:04:03.349780 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.349756    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-95d6c2-dockercfg-99tpw\""
Sep 11 22:04:03.349888 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.349784    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-95d6c2\""
Sep 11 22:04:03.360810 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.360791    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"]
Sep 11 22:04:03.387066 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.387012    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"]
Sep 11 22:04:03.390123 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.390109    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"
Sep 11 22:04:03.401060 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.401039    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"]
Sep 11 22:04:03.422528 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.422291    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nbdm\" (UniqueName: \"kubernetes.io/projected/b58d6418-db5f-41c3-9908-1547d5f7cdec-kube-api-access-4nbdm\") pod \"payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp\" (UID: \"b58d6418-db5f-41c3-9908-1547d5f7cdec\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"
Sep 11 22:04:03.422528 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.422447    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58d6418-db5f-41c3-9908-1547d5f7cdec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp\" (UID: \"b58d6418-db5f-41c3-9908-1547d5f7cdec\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"
Sep 11 22:04:03.523364 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.523294    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bdwd\" (UniqueName: \"kubernetes.io/projected/1f436254-0576-490b-9d08-9968d66f3b85-kube-api-access-4bdwd\") pod \"payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s\" (UID: \"1f436254-0576-490b-9d08-9968d66f3b85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"
Sep 11 22:04:03.523548 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.523377    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f436254-0576-490b-9d08-9968d66f3b85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s\" (UID: \"1f436254-0576-490b-9d08-9968d66f3b85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"
Sep 11 22:04:03.523548 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.523423    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nbdm\" (UniqueName: \"kubernetes.io/projected/b58d6418-db5f-41c3-9908-1547d5f7cdec-kube-api-access-4nbdm\") pod \"payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp\" (UID: \"b58d6418-db5f-41c3-9908-1547d5f7cdec\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"
Sep 11 22:04:03.523667 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.523535    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58d6418-db5f-41c3-9908-1547d5f7cdec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp\" (UID: \"b58d6418-db5f-41c3-9908-1547d5f7cdec\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"
Sep 11 22:04:03.524248 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.524222    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58d6418-db5f-41c3-9908-1547d5f7cdec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp\" (UID: \"b58d6418-db5f-41c3-9908-1547d5f7cdec\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"
Sep 11 22:04:03.534966 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.534940    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nbdm\" (UniqueName: \"kubernetes.io/projected/b58d6418-db5f-41c3-9908-1547d5f7cdec-kube-api-access-4nbdm\") pod \"payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp\" (UID: \"b58d6418-db5f-41c3-9908-1547d5f7cdec\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"
Sep 11 22:04:03.624625 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.624599    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdwd\" (UniqueName: \"kubernetes.io/projected/1f436254-0576-490b-9d08-9968d66f3b85-kube-api-access-4bdwd\") pod \"payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s\" (UID: \"1f436254-0576-490b-9d08-9968d66f3b85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"
Sep 11 22:04:03.625060 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.624631    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f436254-0576-490b-9d08-9968d66f3b85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s\" (UID: \"1f436254-0576-490b-9d08-9968d66f3b85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"
Sep 11 22:04:03.625277 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.625258    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f436254-0576-490b-9d08-9968d66f3b85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s\" (UID: \"1f436254-0576-490b-9d08-9968d66f3b85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"
Sep 11 22:04:03.633084 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.633062    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bdwd\" (UniqueName: \"kubernetes.io/projected/1f436254-0576-490b-9d08-9968d66f3b85-kube-api-access-4bdwd\") pod \"payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s\" (UID: \"1f436254-0576-490b-9d08-9968d66f3b85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"
Sep 11 22:04:03.659818 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.659754    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"
Sep 11 22:04:03.700971 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.700174    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"
Sep 11 22:04:03.729533 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.717292    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:03.780434 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.780402    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:03.935143 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.935119    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"]
Sep 11 22:04:03.936936 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:04:03.936912    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58d6418_db5f_41c3_9908_1547d5f7cdec.slice/crio-34ea38a6591558cf09c9f0f59e484dc0329153d4bc913c839b1ff0e5566c7f39 WatchSource:0}: Error finding container 34ea38a6591558cf09c9f0f59e484dc0329153d4bc913c839b1ff0e5566c7f39: Status 404 returned error can't find the container with id 34ea38a6591558cf09c9f0f59e484dc0329153d4bc913c839b1ff0e5566c7f39
Sep 11 22:04:03.939873 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.939852    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:03.997944 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:03.997921    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"]
Sep 11 22:04:04.000635 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:04:04.000599    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f436254_0576_490b_9d08_9968d66f3b85.slice/crio-f98158c2037fbcfc1f3db587f02f164a349d2ba5a2b4742692edd2830aa46a53 WatchSource:0}: Error finding container f98158c2037fbcfc1f3db587f02f164a349d2ba5a2b4742692edd2830aa46a53: Status 404 returned error can't find the container with id f98158c2037fbcfc1f3db587f02f164a349d2ba5a2b4742692edd2830aa46a53
Sep 11 22:04:04.291583 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.291489    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp" event={"ID":"b58d6418-db5f-41c3-9908-1547d5f7cdec","Type":"ContainerStarted","Data":"1b1221bebe5bced4989fa05b8a9c161915c27a11647793c500acad80d68490df"}
Sep 11 22:04:04.291583 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.291553    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp" event={"ID":"b58d6418-db5f-41c3-9908-1547d5f7cdec","Type":"ContainerStarted","Data":"34ea38a6591558cf09c9f0f59e484dc0329153d4bc913c839b1ff0e5566c7f39"}
Sep 11 22:04:04.291802 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.291646    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"
Sep 11 22:04:04.293160 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.293103    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s" event={"ID":"1f436254-0576-490b-9d08-9968d66f3b85","Type":"ContainerStarted","Data":"5703eecc0fc57b914dcf5ad24a23ab77827176ff91cb2f3d9b74e4c0f1d102f8"}
Sep 11 22:04:04.293321 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.293181    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s" event={"ID":"1f436254-0576-490b-9d08-9968d66f3b85","Type":"ContainerStarted","Data":"f98158c2037fbcfc1f3db587f02f164a349d2ba5a2b4742692edd2830aa46a53"}
Sep 11 22:04:04.293400 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.293374    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"
Sep 11 22:04:04.301196 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.301174    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:04.319115 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.318842    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp" podStartSLOduration=1.318822602 podStartE2EDuration="1.318822602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:04.312302024 +0000 UTC m=+711.047413371" watchObservedRunningTime="2026-09-11 22:04:04.318822602 +0000 UTC m=+711.053933953"
Sep 11 22:04:04.366477 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.366443    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:04.412900 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.412829    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s" podStartSLOduration=1.412806982 podStartE2EDuration="1.412806982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:04.334533953 +0000 UTC m=+711.069645301" watchObservedRunningTime="2026-09-11 22:04:04.412806982 +0000 UTC m=+711.147918331"
Sep 11 22:04:04.415217 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.415195    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"]
Sep 11 22:04:04.419337 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.419317    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"
Sep 11 22:04:04.422914 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.422542    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-27dad9-dockercfg-95lkc\""
Sep 11 22:04:04.422914 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.422547    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-27dad9\""
Sep 11 22:04:04.428081 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.427575    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"]
Sep 11 22:04:04.536957 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.536916    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7x5g\" (UniqueName: \"kubernetes.io/projected/1e375557-c5d4-4752-b35a-69dbfcfc82a2-kube-api-access-k7x5g\") pod \"payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn\" (UID: \"1e375557-c5d4-4752-b35a-69dbfcfc82a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"
Sep 11 22:04:04.537149 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.537088    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e375557-c5d4-4752-b35a-69dbfcfc82a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn\" (UID: \"1e375557-c5d4-4752-b35a-69dbfcfc82a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"
Sep 11 22:04:04.638602 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.638569    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e375557-c5d4-4752-b35a-69dbfcfc82a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn\" (UID: \"1e375557-c5d4-4752-b35a-69dbfcfc82a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"
Sep 11 22:04:04.639084 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.638635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7x5g\" (UniqueName: \"kubernetes.io/projected/1e375557-c5d4-4752-b35a-69dbfcfc82a2-kube-api-access-k7x5g\") pod \"payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn\" (UID: \"1e375557-c5d4-4752-b35a-69dbfcfc82a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"
Sep 11 22:04:04.639443 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.639420    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e375557-c5d4-4752-b35a-69dbfcfc82a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn\" (UID: \"1e375557-c5d4-4752-b35a-69dbfcfc82a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"
Sep 11 22:04:04.646752 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.646727    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7x5g\" (UniqueName: \"kubernetes.io/projected/1e375557-c5d4-4752-b35a-69dbfcfc82a2-kube-api-access-k7x5g\") pod \"payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn\" (UID: \"1e375557-c5d4-4752-b35a-69dbfcfc82a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"
Sep 11 22:04:04.706450 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.706398    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:04.735492 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:04.735458    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"
Sep 11 22:04:05.015986 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:05.015958    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"]
Sep 11 22:04:05.019394 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:04:05.019361    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e375557_c5d4_4752_b35a_69dbfcfc82a2.slice/crio-34ca9a1df2a32020982b664b54003a5bae6412925e2c772d2bde67e8e2545775 WatchSource:0}: Error finding container 34ca9a1df2a32020982b664b54003a5bae6412925e2c772d2bde67e8e2545775: Status 404 returned error can't find the container with id 34ca9a1df2a32020982b664b54003a5bae6412925e2c772d2bde67e8e2545775
Sep 11 22:04:05.155606 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:05.155550    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:05.256362 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:05.256326    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:05.301417 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:05.301337    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn" event={"ID":"1e375557-c5d4-4752-b35a-69dbfcfc82a2","Type":"ContainerStarted","Data":"f671cc5d207baf3f4f801e8dfc78b9638d85b9a44db7f2c24936dea2a9167c81"}
Sep 11 22:04:05.301540 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:05.301430    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"
Sep 11 22:04:05.301540 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:05.301470    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn" event={"ID":"1e375557-c5d4-4752-b35a-69dbfcfc82a2","Type":"ContainerStarted","Data":"34ca9a1df2a32020982b664b54003a5bae6412925e2c772d2bde67e8e2545775"}
Sep 11 22:04:05.319014 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:05.318967    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn" podStartSLOduration=1.3189519760000001 podStartE2EDuration="1.318951976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:05.316700014 +0000 UTC m=+712.051811377" watchObservedRunningTime="2026-09-11 22:04:05.318951976 +0000 UTC m=+712.054063321"
Sep 11 22:04:05.438077 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:05.438032    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:05.532852 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:05.532819    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:05.632555 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:05.632512    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:05.712345 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:05.712291    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:05.780362 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:05.780327    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:05.872730 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:05.872684    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:06.328863 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:06.328826    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:06.429726 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:06.429689    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:08.243566 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:08.243525    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:08.816711 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:08.816675    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:08.917625 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:08.917592    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:09.514674 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:09.514635    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:09.613258 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:09.613222    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:10.016452 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:10.016411    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:11.185904 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:11.185870    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:12.314054 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:12.314021    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:13.663526 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:13.663485    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:13.763836 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:13.763804    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:14.073453 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.073374    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:14.278674 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.278638    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:14.295046 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.295019    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"
Sep 11 22:04:14.295177 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.295150    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"
Sep 11 22:04:14.378629 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.378597    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:14.488818 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.488788    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"]
Sep 11 22:04:14.492552 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.492535    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"
Sep 11 22:04:14.496677 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.496662    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-7d5ec8-dockercfg-jjm92\""
Sep 11 22:04:14.496789 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.496769    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-7d5ec8\""
Sep 11 22:04:14.525692 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.525667    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"]
Sep 11 22:04:14.580861 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.580830    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"]
Sep 11 22:04:14.584467 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.584447    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"
Sep 11 22:04:14.608029 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.608008    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"]
Sep 11 22:04:14.631594 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.631535    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2246aba-6a16-4848-9509-5fa42eb93d3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr\" (UID: \"c2246aba-6a16-4848-9509-5fa42eb93d3b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"
Sep 11 22:04:14.631720 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.631594    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbsj5\" (UniqueName: \"kubernetes.io/projected/c2246aba-6a16-4848-9509-5fa42eb93d3b-kube-api-access-rbsj5\") pod \"payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr\" (UID: \"c2246aba-6a16-4848-9509-5fa42eb93d3b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"
Sep 11 22:04:14.732984 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.732945    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2246aba-6a16-4848-9509-5fa42eb93d3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr\" (UID: \"c2246aba-6a16-4848-9509-5fa42eb93d3b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"
Sep 11 22:04:14.733434 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.733011    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wl9k\" (UniqueName: \"kubernetes.io/projected/d5aed87d-88ba-46ff-93e2-d2a658f32b62-kube-api-access-9wl9k\") pod \"payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d\" (UID: \"d5aed87d-88ba-46ff-93e2-d2a658f32b62\") " pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"
Sep 11 22:04:14.733434 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.733059    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbsj5\" (UniqueName: \"kubernetes.io/projected/c2246aba-6a16-4848-9509-5fa42eb93d3b-kube-api-access-rbsj5\") pod \"payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr\" (UID: \"c2246aba-6a16-4848-9509-5fa42eb93d3b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"
Sep 11 22:04:14.733434 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.733103    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aed87d-88ba-46ff-93e2-d2a658f32b62-plugins-config-volume\") pod \"payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d\" (UID: \"d5aed87d-88ba-46ff-93e2-d2a658f32b62\") " pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"
Sep 11 22:04:14.733841 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.733820    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2246aba-6a16-4848-9509-5fa42eb93d3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr\" (UID: \"c2246aba-6a16-4848-9509-5fa42eb93d3b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"
Sep 11 22:04:14.741166 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.741148    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbsj5\" (UniqueName: \"kubernetes.io/projected/c2246aba-6a16-4848-9509-5fa42eb93d3b-kube-api-access-rbsj5\") pod \"payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr\" (UID: \"c2246aba-6a16-4848-9509-5fa42eb93d3b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"
Sep 11 22:04:14.803058 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.803039    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"
Sep 11 22:04:14.834722 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.834692    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wl9k\" (UniqueName: \"kubernetes.io/projected/d5aed87d-88ba-46ff-93e2-d2a658f32b62-kube-api-access-9wl9k\") pod \"payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d\" (UID: \"d5aed87d-88ba-46ff-93e2-d2a658f32b62\") " pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"
Sep 11 22:04:14.834841 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.834780    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aed87d-88ba-46ff-93e2-d2a658f32b62-plugins-config-volume\") pod \"payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d\" (UID: \"d5aed87d-88ba-46ff-93e2-d2a658f32b62\") " pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"
Sep 11 22:04:14.835453 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.835430    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aed87d-88ba-46ff-93e2-d2a658f32b62-plugins-config-volume\") pod \"payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d\" (UID: \"d5aed87d-88ba-46ff-93e2-d2a658f32b62\") " pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"
Sep 11 22:04:14.842522 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.842503    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wl9k\" (UniqueName: \"kubernetes.io/projected/d5aed87d-88ba-46ff-93e2-d2a658f32b62-kube-api-access-9wl9k\") pod \"payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d\" (UID: \"d5aed87d-88ba-46ff-93e2-d2a658f32b62\") " pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"
Sep 11 22:04:14.894754 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.894680    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"
Sep 11 22:04:14.951285 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:14.950963    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:15.052245 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.052211    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:15.074458 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.074430    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"]
Sep 11 22:04:15.077405 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:04:15.077378    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2246aba_6a16_4848_9509_5fa42eb93d3b.slice/crio-904a2a932dd5c565f3a5950764894b45b6a30c25c43ac5448b83c335a2d63682 WatchSource:0}: Error finding container 904a2a932dd5c565f3a5950764894b45b6a30c25c43ac5448b83c335a2d63682: Status 404 returned error can't find the container with id 904a2a932dd5c565f3a5950764894b45b6a30c25c43ac5448b83c335a2d63682
Sep 11 22:04:15.177720 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.177694    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"]
Sep 11 22:04:15.179046 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:04:15.179016    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5aed87d_88ba_46ff_93e2_d2a658f32b62.slice/crio-2053ca28854de5591fefeab1546a10a315f34d998945808b805bc784c8511b6f WatchSource:0}: Error finding container 2053ca28854de5591fefeab1546a10a315f34d998945808b805bc784c8511b6f: Status 404 returned error can't find the container with id 2053ca28854de5591fefeab1546a10a315f34d998945808b805bc784c8511b6f
Sep 11 22:04:15.303235 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.303206    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"
Sep 11 22:04:15.303589 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.303565    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"
Sep 11 22:04:15.359275 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.359238    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d" event={"ID":"d5aed87d-88ba-46ff-93e2-d2a658f32b62","Type":"ContainerStarted","Data":"1152859c2f36d2c326e152d170c7cac23dc3866a2123314ab30d753ad6cbf625"}
Sep 11 22:04:15.359476 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.359297    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d" event={"ID":"d5aed87d-88ba-46ff-93e2-d2a658f32b62","Type":"ContainerStarted","Data":"2053ca28854de5591fefeab1546a10a315f34d998945808b805bc784c8511b6f"}
Sep 11 22:04:15.359782 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.359737    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"
Sep 11 22:04:15.361699 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.361676    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr" event={"ID":"c2246aba-6a16-4848-9509-5fa42eb93d3b","Type":"ContainerStarted","Data":"581bd249bc5a69e882730d1783a40f48b4783f47fba33695d7f9657e1fa65a5a"}
Sep 11 22:04:15.361810 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.361713    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr" event={"ID":"c2246aba-6a16-4848-9509-5fa42eb93d3b","Type":"ContainerStarted","Data":"904a2a932dd5c565f3a5950764894b45b6a30c25c43ac5448b83c335a2d63682"}
Sep 11 22:04:15.361882 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.361827    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"
Sep 11 22:04:15.364924 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.364894    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:15.378060 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.378011    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d" podStartSLOduration=1.377993572 podStartE2EDuration="1.377993572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:15.376300566 +0000 UTC m=+722.111411914" watchObservedRunningTime="2026-09-11 22:04:15.377993572 +0000 UTC m=+722.113104920"
Sep 11 22:04:15.396643 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.396606    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr" podStartSLOduration=1.396592833 podStartE2EDuration="1.396592833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:15.394673619 +0000 UTC m=+722.129784976" watchObservedRunningTime="2026-09-11 22:04:15.396592833 +0000 UTC m=+722.131704178"
Sep 11 22:04:15.628791 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.628753    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:15.737514 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.737461    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:15.785061 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.785028    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"]
Sep 11 22:04:15.789056 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.789034    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"
Sep 11 22:04:15.792460 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.792253    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f823e45b\""
Sep 11 22:04:15.792460 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.792300    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f823e45b-dockercfg-qhxtg\""
Sep 11 22:04:15.801796 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.801774    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"]
Sep 11 22:04:15.853639 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.853606    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:15.947230 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.947151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtgcp\" (UniqueName: \"kubernetes.io/projected/148979fe-1cf0-48c4-9f51-dbd691b0f82c-kube-api-access-xtgcp\") pod \"payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx\" (UID: \"148979fe-1cf0-48c4-9f51-dbd691b0f82c\") " pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"
Sep 11 22:04:15.947424 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.947372    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148979fe-1cf0-48c4-9f51-dbd691b0f82c-plugins-config-volume\") pod \"payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx\" (UID: \"148979fe-1cf0-48c4-9f51-dbd691b0f82c\") " pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"
Sep 11 22:04:15.951500 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:15.951475    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:16.042377 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:16.042336    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:16.048512 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:16.048484    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtgcp\" (UniqueName: \"kubernetes.io/projected/148979fe-1cf0-48c4-9f51-dbd691b0f82c-kube-api-access-xtgcp\") pod \"payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx\" (UID: \"148979fe-1cf0-48c4-9f51-dbd691b0f82c\") " pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"
Sep 11 22:04:16.048658 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:16.048608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148979fe-1cf0-48c4-9f51-dbd691b0f82c-plugins-config-volume\") pod \"payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx\" (UID: \"148979fe-1cf0-48c4-9f51-dbd691b0f82c\") " pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"
Sep 11 22:04:16.049339 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:16.049302    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148979fe-1cf0-48c4-9f51-dbd691b0f82c-plugins-config-volume\") pod \"payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx\" (UID: \"148979fe-1cf0-48c4-9f51-dbd691b0f82c\") " pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"
Sep 11 22:04:16.058838 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:16.058817    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtgcp\" (UniqueName: \"kubernetes.io/projected/148979fe-1cf0-48c4-9f51-dbd691b0f82c-kube-api-access-xtgcp\") pod \"payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx\" (UID: \"148979fe-1cf0-48c4-9f51-dbd691b0f82c\") " pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"
Sep 11 22:04:16.101027 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:16.100992    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"
Sep 11 22:04:16.110249 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:16.110216    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:16.308582 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:16.308516    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"
Sep 11 22:04:16.376377 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:16.376354    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"]
Sep 11 22:04:16.378238 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:04:16.378211    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148979fe_1cf0_48c4_9f51_dbd691b0f82c.slice/crio-f36d962ad2f293cf3c5d5035377bce3d780e6a74b7fdd97ba91be264249d4800 WatchSource:0}: Error finding container f36d962ad2f293cf3c5d5035377bce3d780e6a74b7fdd97ba91be264249d4800: Status 404 returned error can't find the container with id f36d962ad2f293cf3c5d5035377bce3d780e6a74b7fdd97ba91be264249d4800
Sep 11 22:04:16.450326 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:16.450277    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:16.691818 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:16.691779    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:17.309884 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:17.309843    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:17.373218 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:17.373178    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx" event={"ID":"148979fe-1cf0-48c4-9f51-dbd691b0f82c","Type":"ContainerStarted","Data":"86eedbedd3a968106e7e68c966f48e1b7950d2a973eab703d20224b95b9315f1"}
Sep 11 22:04:17.373411 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:17.373276    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx" event={"ID":"148979fe-1cf0-48c4-9f51-dbd691b0f82c","Type":"ContainerStarted","Data":"f36d962ad2f293cf3c5d5035377bce3d780e6a74b7fdd97ba91be264249d4800"}
Sep 11 22:04:17.373411 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:17.373323    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"
Sep 11 22:04:17.391763 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:17.391703    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx" podStartSLOduration=2.391675136 podStartE2EDuration="2.391675136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:17.389441209 +0000 UTC m=+724.124552557" watchObservedRunningTime="2026-09-11 22:04:17.391675136 +0000 UTC m=+724.126786485"
Sep 11 22:04:18.533451 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:18.533418    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:18.695293 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:18.695256    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:19.601033 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:19.601001    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:19.758733 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:19.758702    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:20.121187 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:20.121154    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:20.439958 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:20.439862    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:20.544604 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:20.544571    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:20.640072 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:20.640037    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:20.741864 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:20.741756    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:20.843569 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:20.843533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:20.949375 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:20.949344    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:21.055768 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:21.055691    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:21.147895 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:21.147858    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:21.748732 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:21.748696    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:22.625814 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:22.625776    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:23.724452 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:23.724414    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:24.631173 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:24.631140    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:24.731280 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:24.731244    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:25.199276 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:25.199242    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:25.304384 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:25.304348    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:25.736919 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:25.736882    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:25.909857 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:25.909813    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:26.369437 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:26.369403    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"
Sep 11 22:04:26.369613 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:26.369586    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"
Sep 11 22:04:26.546212 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:26.546178    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:27.349428 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:27.349394    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:28.205514 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:28.205484    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:28.383509 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:28.383484    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"
Sep 11 22:04:32.586347 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:32.586290    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:32.698842 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:32.698808    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:32.811581 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:32.811546    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:34.838662 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:34.838623    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:36.284205 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:36.284165    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:37.005918 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:37.005878    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:37.893251 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:37.893216    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:38.776067 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:38.776030    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:38.882565 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:38.882531    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:38.977616 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:38.977584    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:39.188859 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:39.188824    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:39.332602 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:39.332575    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:39.448962 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:39.448894    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf"]
Sep 11 22:04:39.452420 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:39.452404    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf"
Sep 11 22:04:39.455078 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:39.455060    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:04:39.455180 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:39.455092    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:04:39.460981 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:39.460958    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf"]
Sep 11 22:04:39.515767 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:39.515739    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:39.575159 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:39.575136    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e94e003-9fc1-4ee3-99c1-02275aebc297-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf\" (UID: \"7e94e003-9fc1-4ee3-99c1-02275aebc297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf"
Sep 11 22:04:39.675864 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:39.675836    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e94e003-9fc1-4ee3-99c1-02275aebc297-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf\" (UID: \"7e94e003-9fc1-4ee3-99c1-02275aebc297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf"
Sep 11 22:04:39.676460 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:39.676441    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e94e003-9fc1-4ee3-99c1-02275aebc297-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf\" (UID: \"7e94e003-9fc1-4ee3-99c1-02275aebc297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf"
Sep 11 22:04:39.766080 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:39.765984    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf"
Sep 11 22:04:39.787646 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:39.787619    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:40.042543 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:40.042521    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf"]
Sep 11 22:04:40.044845 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:04:40.044815    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e94e003_9fc1_4ee3_99c1_02275aebc297.slice/crio-8e0a2e8908001f755655a5b7a22ef3b6cd746705daf52c8a153a7c4b2ec2d825 WatchSource:0}: Error finding container 8e0a2e8908001f755655a5b7a22ef3b6cd746705daf52c8a153a7c4b2ec2d825: Status 404 returned error can't find the container with id 8e0a2e8908001f755655a5b7a22ef3b6cd746705daf52c8a153a7c4b2ec2d825
Sep 11 22:04:40.502545 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:40.502514    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf" event={"ID":"7e94e003-9fc1-4ee3-99c1-02275aebc297","Type":"ContainerStarted","Data":"8e0a2e8908001f755655a5b7a22ef3b6cd746705daf52c8a153a7c4b2ec2d825"}
Sep 11 22:04:40.707563 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:40.707532    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:41.472869 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:41.472817    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:42.210412 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:42.210372    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:42.857972 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:42.857946    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:43.504333 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:43.504219    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:43.522184 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:43.522156    2574 generic.go:356] "Generic (PLEG): container finished" podID="7e94e003-9fc1-4ee3-99c1-02275aebc297" containerID="1cf10301aae63bf106539d9ad5d303c3b300ee53d1f41fe3609e91f6c16c8bdf" exitCode=0
Sep 11 22:04:43.522357 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:43.522228    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf" event={"ID":"7e94e003-9fc1-4ee3-99c1-02275aebc297","Type":"ContainerDied","Data":"1cf10301aae63bf106539d9ad5d303c3b300ee53d1f41fe3609e91f6c16c8bdf"}
Sep 11 22:04:44.414426 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:44.414384    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:44.659402 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:44.659377    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf"
Sep 11 22:04:44.722144 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:44.722073    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e94e003-9fc1-4ee3-99c1-02275aebc297-maas-api-service-ca\") pod \"7e94e003-9fc1-4ee3-99c1-02275aebc297\" (UID: \"7e94e003-9fc1-4ee3-99c1-02275aebc297\") "
Sep 11 22:04:44.722424 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:44.722401    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e94e003-9fc1-4ee3-99c1-02275aebc297-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e94e003-9fc1-4ee3-99c1-02275aebc297" (UID: "7e94e003-9fc1-4ee3-99c1-02275aebc297"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:04:44.823228 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:44.823199    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e94e003-9fc1-4ee3-99c1-02275aebc297-maas-api-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:04:45.101927 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:45.101888    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:45.258385 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:45.258341    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:45.365916 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:45.365832    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:45.535726 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:45.535702    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf"
Sep 11 22:04:45.536095 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:45.535687    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf" event={"ID":"7e94e003-9fc1-4ee3-99c1-02275aebc297","Type":"ContainerDied","Data":"8e0a2e8908001f755655a5b7a22ef3b6cd746705daf52c8a153a7c4b2ec2d825"}
Sep 11 22:04:45.536095 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:45.535828    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e0a2e8908001f755655a5b7a22ef3b6cd746705daf52c8a153a7c4b2ec2d825"
Sep 11 22:04:45.937221 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:45.937174    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:50.419726 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:50.419671    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:50.985272 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:50.985240    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:51.012259 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.012222    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"]
Sep 11 22:04:51.012487 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.012459    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx" podUID="148979fe-1cf0-48c4-9f51-dbd691b0f82c" containerName="payload-processing" containerID="cri-o://86eedbedd3a968106e7e68c966f48e1b7950d2a973eab703d20224b95b9315f1" gracePeriod=30
Sep 11 22:04:51.082454 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.082421    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:51.174750 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.174714    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf"]
Sep 11 22:04:51.175912 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.175883    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:51.177383 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.177358    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f823e45b-m6jlf"]
Sep 11 22:04:51.238364 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.238281    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:51.279666 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.279645    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"
Sep 11 22:04:51.339239 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.339200    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:51.384809 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.384786    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtgcp\" (UniqueName: \"kubernetes.io/projected/148979fe-1cf0-48c4-9f51-dbd691b0f82c-kube-api-access-xtgcp\") pod \"148979fe-1cf0-48c4-9f51-dbd691b0f82c\" (UID: \"148979fe-1cf0-48c4-9f51-dbd691b0f82c\") "
Sep 11 22:04:51.384966 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.384820    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148979fe-1cf0-48c4-9f51-dbd691b0f82c-plugins-config-volume\") pod \"148979fe-1cf0-48c4-9f51-dbd691b0f82c\" (UID: \"148979fe-1cf0-48c4-9f51-dbd691b0f82c\") "
Sep 11 22:04:51.385214 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.385189    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148979fe-1cf0-48c4-9f51-dbd691b0f82c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "148979fe-1cf0-48c4-9f51-dbd691b0f82c" (UID: "148979fe-1cf0-48c4-9f51-dbd691b0f82c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:04:51.386991 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.386967    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148979fe-1cf0-48c4-9f51-dbd691b0f82c-kube-api-access-xtgcp" (OuterVolumeSpecName: "kube-api-access-xtgcp") pod "148979fe-1cf0-48c4-9f51-dbd691b0f82c" (UID: "148979fe-1cf0-48c4-9f51-dbd691b0f82c"). InnerVolumeSpecName "kube-api-access-xtgcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:04:51.486239 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.486212    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtgcp\" (UniqueName: \"kubernetes.io/projected/148979fe-1cf0-48c4-9f51-dbd691b0f82c-kube-api-access-xtgcp\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:04:51.486239 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.486238    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148979fe-1cf0-48c4-9f51-dbd691b0f82c-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:04:51.574372 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.574277    2574 generic.go:356] "Generic (PLEG): container finished" podID="148979fe-1cf0-48c4-9f51-dbd691b0f82c" containerID="86eedbedd3a968106e7e68c966f48e1b7950d2a973eab703d20224b95b9315f1" exitCode=0
Sep 11 22:04:51.574372 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.574346    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"
Sep 11 22:04:51.574531 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.574361    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx" event={"ID":"148979fe-1cf0-48c4-9f51-dbd691b0f82c","Type":"ContainerDied","Data":"86eedbedd3a968106e7e68c966f48e1b7950d2a973eab703d20224b95b9315f1"}
Sep 11 22:04:51.574531 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.574437    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx" event={"ID":"148979fe-1cf0-48c4-9f51-dbd691b0f82c","Type":"ContainerDied","Data":"f36d962ad2f293cf3c5d5035377bce3d780e6a74b7fdd97ba91be264249d4800"}
Sep 11 22:04:51.574531 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.574460    2574 scope.go:117] "RemoveContainer" containerID="86eedbedd3a968106e7e68c966f48e1b7950d2a973eab703d20224b95b9315f1"
Sep 11 22:04:51.583597 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.583582    2574 scope.go:117] "RemoveContainer" containerID="86eedbedd3a968106e7e68c966f48e1b7950d2a973eab703d20224b95b9315f1"
Sep 11 22:04:51.583868 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:04:51.583850    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86eedbedd3a968106e7e68c966f48e1b7950d2a973eab703d20224b95b9315f1\": container with ID starting with 86eedbedd3a968106e7e68c966f48e1b7950d2a973eab703d20224b95b9315f1 not found: ID does not exist" containerID="86eedbedd3a968106e7e68c966f48e1b7950d2a973eab703d20224b95b9315f1"
Sep 11 22:04:51.583928 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.583876    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86eedbedd3a968106e7e68c966f48e1b7950d2a973eab703d20224b95b9315f1"} err="failed to get container status \"86eedbedd3a968106e7e68c966f48e1b7950d2a973eab703d20224b95b9315f1\": rpc error: code = NotFound desc = could not find container \"86eedbedd3a968106e7e68c966f48e1b7950d2a973eab703d20224b95b9315f1\": container with ID starting with 86eedbedd3a968106e7e68c966f48e1b7950d2a973eab703d20224b95b9315f1 not found: ID does not exist"
Sep 11 22:04:51.596756 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.596732    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"]
Sep 11 22:04:51.600340 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.600304    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f823e45b-795d4976bf-dfxtx"]
Sep 11 22:04:51.613864 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.613831    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:51.722206 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.722171    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="148979fe-1cf0-48c4-9f51-dbd691b0f82c" path="/var/lib/kubelet/pods/148979fe-1cf0-48c4-9f51-dbd691b0f82c/volumes"
Sep 11 22:04:51.722668 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.722647    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e94e003-9fc1-4ee3-99c1-02275aebc297" path="/var/lib/kubelet/pods/7e94e003-9fc1-4ee3-99c1-02275aebc297/volumes"
Sep 11 22:04:51.820156 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.820121    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:51.919677 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:51.919643    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:52.040629 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:52.040595    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:52.371092 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:52.371064    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:52.443180 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:52.443128    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:52.557859 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:52.557829    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:53.198408 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:53.198377    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:53.996362 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:53.996296    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:54.901527 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:54.901498    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:55.244365 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:55.244277    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:55.340554 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:55.340521    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:55.715762 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:55.715728    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:56.403288 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.403234    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:56.466111 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.466080    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:56.564180 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.564141    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:56.849414 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.849368    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:56.949352 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.949295    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:56.965681 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.965655    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"]
Sep 11 22:04:56.966096 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.966084    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e94e003-9fc1-4ee3-99c1-02275aebc297" containerName="revoke-keys"
Sep 11 22:04:56.966150 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.966097    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e94e003-9fc1-4ee3-99c1-02275aebc297" containerName="revoke-keys"
Sep 11 22:04:56.966150 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.966108    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="148979fe-1cf0-48c4-9f51-dbd691b0f82c" containerName="payload-processing"
Sep 11 22:04:56.966150 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.966114    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="148979fe-1cf0-48c4-9f51-dbd691b0f82c" containerName="payload-processing"
Sep 11 22:04:56.966258 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.966195    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="148979fe-1cf0-48c4-9f51-dbd691b0f82c" containerName="payload-processing"
Sep 11 22:04:56.966258 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.966206    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e94e003-9fc1-4ee3-99c1-02275aebc297" containerName="revoke-keys"
Sep 11 22:04:56.970748 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.970732    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"
Sep 11 22:04:56.973259 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.973242    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-0cd94651\""
Sep 11 22:04:56.982349 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.982330    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"]
Sep 11 22:04:56.992675 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.992655    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"]
Sep 11 22:04:56.998987 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:56.998958    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"
Sep 11 22:04:57.003589 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.003565    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-0cd94651-dockercfg-85fsn\""
Sep 11 22:04:57.009500 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.009374    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"]
Sep 11 22:04:57.142102 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.142019    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vb8m\" (UniqueName: \"kubernetes.io/projected/87207e93-7303-4c9a-bbd9-cd7107e296c3-kube-api-access-9vb8m\") pod \"payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv\" (UID: \"87207e93-7303-4c9a-bbd9-cd7107e296c3\") " pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"
Sep 11 22:04:57.142102 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.142078    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ec7893-db2a-4420-8d5e-03b6322687a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7\" (UID: \"d2ec7893-db2a-4420-8d5e-03b6322687a7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"
Sep 11 22:04:57.142283 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.142159    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drt5c\" (UniqueName: \"kubernetes.io/projected/d2ec7893-db2a-4420-8d5e-03b6322687a7-kube-api-access-drt5c\") pod \"payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7\" (UID: \"d2ec7893-db2a-4420-8d5e-03b6322687a7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"
Sep 11 22:04:57.142283 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.142235    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87207e93-7303-4c9a-bbd9-cd7107e296c3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv\" (UID: \"87207e93-7303-4c9a-bbd9-cd7107e296c3\") " pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"
Sep 11 22:04:57.243364 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.243302    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87207e93-7303-4c9a-bbd9-cd7107e296c3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv\" (UID: \"87207e93-7303-4c9a-bbd9-cd7107e296c3\") " pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"
Sep 11 22:04:57.243523 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.243391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vb8m\" (UniqueName: \"kubernetes.io/projected/87207e93-7303-4c9a-bbd9-cd7107e296c3-kube-api-access-9vb8m\") pod \"payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv\" (UID: \"87207e93-7303-4c9a-bbd9-cd7107e296c3\") " pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"
Sep 11 22:04:57.243523 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.243454    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ec7893-db2a-4420-8d5e-03b6322687a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7\" (UID: \"d2ec7893-db2a-4420-8d5e-03b6322687a7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"
Sep 11 22:04:57.243523 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.243504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drt5c\" (UniqueName: \"kubernetes.io/projected/d2ec7893-db2a-4420-8d5e-03b6322687a7-kube-api-access-drt5c\") pod \"payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7\" (UID: \"d2ec7893-db2a-4420-8d5e-03b6322687a7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"
Sep 11 22:04:57.244115 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.244095    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87207e93-7303-4c9a-bbd9-cd7107e296c3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv\" (UID: \"87207e93-7303-4c9a-bbd9-cd7107e296c3\") " pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"
Sep 11 22:04:57.244243 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.244226    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ec7893-db2a-4420-8d5e-03b6322687a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7\" (UID: \"d2ec7893-db2a-4420-8d5e-03b6322687a7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"
Sep 11 22:04:57.257755 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.257723    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drt5c\" (UniqueName: \"kubernetes.io/projected/d2ec7893-db2a-4420-8d5e-03b6322687a7-kube-api-access-drt5c\") pod \"payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7\" (UID: \"d2ec7893-db2a-4420-8d5e-03b6322687a7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"
Sep 11 22:04:57.258356 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.258334    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vb8m\" (UniqueName: \"kubernetes.io/projected/87207e93-7303-4c9a-bbd9-cd7107e296c3-kube-api-access-9vb8m\") pod \"payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv\" (UID: \"87207e93-7303-4c9a-bbd9-cd7107e296c3\") " pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"
Sep 11 22:04:57.282586 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.282567    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"
Sep 11 22:04:57.313646 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.313622    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"
Sep 11 22:04:57.396893 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.396788    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:57.560661 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.560636    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"]
Sep 11 22:04:57.562505 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:04:57.562480    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ec7893_db2a_4420_8d5e_03b6322687a7.slice/crio-91633d46cdc827219da4a03be5c990298a34764723fc303aed97e7382d583c21 WatchSource:0}: Error finding container 91633d46cdc827219da4a03be5c990298a34764723fc303aed97e7382d583c21: Status 404 returned error can't find the container with id 91633d46cdc827219da4a03be5c990298a34764723fc303aed97e7382d583c21
Sep 11 22:04:57.602489 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.602465    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"]
Sep 11 22:04:57.603702 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:04:57.603665    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87207e93_7303_4c9a_bbd9_cd7107e296c3.slice/crio-8b78803820bd292a4de71a1c655d39d298a277c3b9069437dade01286b356da9 WatchSource:0}: Error finding container 8b78803820bd292a4de71a1c655d39d298a277c3b9069437dade01286b356da9: Status 404 returned error can't find the container with id 8b78803820bd292a4de71a1c655d39d298a277c3b9069437dade01286b356da9
Sep 11 22:04:57.608592 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:57.608568    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7" event={"ID":"d2ec7893-db2a-4420-8d5e-03b6322687a7","Type":"ContainerStarted","Data":"91633d46cdc827219da4a03be5c990298a34764723fc303aed97e7382d583c21"}
Sep 11 22:04:58.154593 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:58.154557    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:58.617076 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:58.617039    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv" event={"ID":"87207e93-7303-4c9a-bbd9-cd7107e296c3","Type":"ContainerStarted","Data":"a42f17e72f002e44f4d7c884a86829e9c62fd79d0137bbfa82a2d17f957e26da"}
Sep 11 22:04:58.617573 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:58.617111    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"
Sep 11 22:04:58.617573 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:58.617136    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv" event={"ID":"87207e93-7303-4c9a-bbd9-cd7107e296c3","Type":"ContainerStarted","Data":"8b78803820bd292a4de71a1c655d39d298a277c3b9069437dade01286b356da9"}
Sep 11 22:04:58.618850 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:58.618821    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7" event={"ID":"d2ec7893-db2a-4420-8d5e-03b6322687a7","Type":"ContainerStarted","Data":"104f3d2d392dff7a3b9d76652a2bad80170f6a0a1398b83c2eb29e1559bb994a"}
Sep 11 22:04:58.618976 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:58.618902    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"
Sep 11 22:04:58.633867 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:58.633812    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv" podStartSLOduration=2.6337926879999998 podStartE2EDuration="2.633792688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:58.632849566 +0000 UTC m=+765.367960908" watchObservedRunningTime="2026-09-11 22:04:58.633792688 +0000 UTC m=+765.368904036"
Sep 11 22:04:58.649925 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:58.649882    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7" podStartSLOduration=2.649869087 podStartE2EDuration="2.649869087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:58.648847951 +0000 UTC m=+765.383959300" watchObservedRunningTime="2026-09-11 22:04:58.649869087 +0000 UTC m=+765.384980461"
Sep 11 22:04:58.822723 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:58.822686    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:59.620327 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:04:59.620270    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:00.277005 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:00.276969    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:03.589279 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:03.589240    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:09.627377 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:09.627340    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"
Sep 11 22:05:09.627802 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:09.627432    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"
Sep 11 22:05:11.195899 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:11.195864    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:18.589614 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:18.589571    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:19.496234 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:19.496183    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:20.173897 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:20.173862    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:20.237685 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:20.237656    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg"]
Sep 11 22:05:20.246528 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:20.246504    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg"
Sep 11 22:05:20.250116 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:20.250082    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg"]
Sep 11 22:05:20.250248 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:20.250144    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:05:20.250248 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:20.250217    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:05:20.351871 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:20.351836    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81ecc2a-4334-4918-8a87-0771282c99c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg\" (UID: \"c81ecc2a-4334-4918-8a87-0771282c99c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg"
Sep 11 22:05:20.452756 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:20.452687    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81ecc2a-4334-4918-8a87-0771282c99c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg\" (UID: \"c81ecc2a-4334-4918-8a87-0771282c99c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg"
Sep 11 22:05:20.453257 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:20.453237    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81ecc2a-4334-4918-8a87-0771282c99c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg\" (UID: \"c81ecc2a-4334-4918-8a87-0771282c99c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg"
Sep 11 22:05:20.560104 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:20.560073    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg"
Sep 11 22:05:20.829462 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:20.829437    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg"]
Sep 11 22:05:20.831892 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:05:20.831860    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81ecc2a_4334_4918_8a87_0771282c99c8.slice/crio-72ffad95dc5a5188888ec8e4f129f6903c131b24b2a4a6dd11295f9f2bb48bfe WatchSource:0}: Error finding container 72ffad95dc5a5188888ec8e4f129f6903c131b24b2a4a6dd11295f9f2bb48bfe: Status 404 returned error can't find the container with id 72ffad95dc5a5188888ec8e4f129f6903c131b24b2a4a6dd11295f9f2bb48bfe
Sep 11 22:05:20.920081 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:20.920052    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:21.718657 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:21.718615    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:21.754282 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:21.754248    2574 generic.go:356] "Generic (PLEG): container finished" podID="c81ecc2a-4334-4918-8a87-0771282c99c8" containerID="941b49060e3e3528cfad9adab1b4e35bcf26052e3b80f4fa1e18203da7caa595" exitCode=0
Sep 11 22:05:21.754422 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:21.754287    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg" event={"ID":"c81ecc2a-4334-4918-8a87-0771282c99c8","Type":"ContainerDied","Data":"941b49060e3e3528cfad9adab1b4e35bcf26052e3b80f4fa1e18203da7caa595"}
Sep 11 22:05:21.754422 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:21.754343    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg" event={"ID":"c81ecc2a-4334-4918-8a87-0771282c99c8","Type":"ContainerStarted","Data":"72ffad95dc5a5188888ec8e4f129f6903c131b24b2a4a6dd11295f9f2bb48bfe"}
Sep 11 22:05:22.403728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:22.403675    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:22.904029 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:22.904006    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg"
Sep 11 22:05:22.975480 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:22.975451    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81ecc2a-4334-4918-8a87-0771282c99c8-maas-api-service-ca\") pod \"c81ecc2a-4334-4918-8a87-0771282c99c8\" (UID: \"c81ecc2a-4334-4918-8a87-0771282c99c8\") "
Sep 11 22:05:22.975823 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:22.975802    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81ecc2a-4334-4918-8a87-0771282c99c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c81ecc2a-4334-4918-8a87-0771282c99c8" (UID: "c81ecc2a-4334-4918-8a87-0771282c99c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:23.077218 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:23.077144    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81ecc2a-4334-4918-8a87-0771282c99c8-maas-api-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:05:23.169229 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:23.169199    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:23.766408 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:23.766380    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg"
Sep 11 22:05:23.766553 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:23.766431    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg" event={"ID":"c81ecc2a-4334-4918-8a87-0771282c99c8","Type":"ContainerDied","Data":"72ffad95dc5a5188888ec8e4f129f6903c131b24b2a4a6dd11295f9f2bb48bfe"}
Sep 11 22:05:23.766553 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:23.766483    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72ffad95dc5a5188888ec8e4f129f6903c131b24b2a4a6dd11295f9f2bb48bfe"
Sep 11 22:05:24.050494 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:24.050409    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:24.728442 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:24.728406    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:25.523046 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:25.523003    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:26.372424 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:26.372391    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:26.956859 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:26.956830    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:26.979573 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:26.979545    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"]
Sep 11 22:05:26.979832 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:26.979803    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv" podUID="87207e93-7303-4c9a-bbd9-cd7107e296c3" containerName="payload-processing" containerID="cri-o://a42f17e72f002e44f4d7c884a86829e9c62fd79d0137bbfa82a2d17f957e26da" gracePeriod=30
Sep 11 22:05:26.989756 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:26.989730    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"]
Sep 11 22:05:26.989994 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:26.989956    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7" podUID="d2ec7893-db2a-4420-8d5e-03b6322687a7" containerName="payload-pre-processing" containerID="cri-o://104f3d2d392dff7a3b9d76652a2bad80170f6a0a1398b83c2eb29e1559bb994a" gracePeriod=30
Sep 11 22:05:27.061044 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.061017    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:27.145906 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.145839    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg"]
Sep 11 22:05:27.149015 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.148980    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:27.150038 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.150013    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0cd94651-l9ckg"]
Sep 11 22:05:27.200545 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.200134    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:27.289094 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.289067    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:27.289819 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.289797    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"
Sep 11 22:05:27.293592 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.293574    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"
Sep 11 22:05:27.416475 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.416436    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ec7893-db2a-4420-8d5e-03b6322687a7-plugins-config-volume\") pod \"d2ec7893-db2a-4420-8d5e-03b6322687a7\" (UID: \"d2ec7893-db2a-4420-8d5e-03b6322687a7\") "
Sep 11 22:05:27.416666 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.416497    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vb8m\" (UniqueName: \"kubernetes.io/projected/87207e93-7303-4c9a-bbd9-cd7107e296c3-kube-api-access-9vb8m\") pod \"87207e93-7303-4c9a-bbd9-cd7107e296c3\" (UID: \"87207e93-7303-4c9a-bbd9-cd7107e296c3\") "
Sep 11 22:05:27.416732 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.416681    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87207e93-7303-4c9a-bbd9-cd7107e296c3-plugins-config-volume\") pod \"87207e93-7303-4c9a-bbd9-cd7107e296c3\" (UID: \"87207e93-7303-4c9a-bbd9-cd7107e296c3\") "
Sep 11 22:05:27.416807 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.416788    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drt5c\" (UniqueName: \"kubernetes.io/projected/d2ec7893-db2a-4420-8d5e-03b6322687a7-kube-api-access-drt5c\") pod \"d2ec7893-db2a-4420-8d5e-03b6322687a7\" (UID: \"d2ec7893-db2a-4420-8d5e-03b6322687a7\") "
Sep 11 22:05:27.416865 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.416809    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ec7893-db2a-4420-8d5e-03b6322687a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2ec7893-db2a-4420-8d5e-03b6322687a7" (UID: "d2ec7893-db2a-4420-8d5e-03b6322687a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:27.417043 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.417019    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87207e93-7303-4c9a-bbd9-cd7107e296c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87207e93-7303-4c9a-bbd9-cd7107e296c3" (UID: "87207e93-7303-4c9a-bbd9-cd7107e296c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:27.417163 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.417102    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ec7893-db2a-4420-8d5e-03b6322687a7-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.417163 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.417115    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87207e93-7303-4c9a-bbd9-cd7107e296c3-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.418772 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.418746    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87207e93-7303-4c9a-bbd9-cd7107e296c3-kube-api-access-9vb8m" (OuterVolumeSpecName: "kube-api-access-9vb8m") pod "87207e93-7303-4c9a-bbd9-cd7107e296c3" (UID: "87207e93-7303-4c9a-bbd9-cd7107e296c3"). InnerVolumeSpecName "kube-api-access-9vb8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:27.418966 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.418942    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ec7893-db2a-4420-8d5e-03b6322687a7-kube-api-access-drt5c" (OuterVolumeSpecName: "kube-api-access-drt5c") pod "d2ec7893-db2a-4420-8d5e-03b6322687a7" (UID: "d2ec7893-db2a-4420-8d5e-03b6322687a7"). InnerVolumeSpecName "kube-api-access-drt5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:27.518440 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.518368    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vb8m\" (UniqueName: \"kubernetes.io/projected/87207e93-7303-4c9a-bbd9-cd7107e296c3-kube-api-access-9vb8m\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.518440 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.518391    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drt5c\" (UniqueName: \"kubernetes.io/projected/d2ec7893-db2a-4420-8d5e-03b6322687a7-kube-api-access-drt5c\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.544706 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.544677    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:27.720745 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.720701    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c81ecc2a-4334-4918-8a87-0771282c99c8" path="/var/lib/kubelet/pods/c81ecc2a-4334-4918-8a87-0771282c99c8/volumes"
Sep 11 22:05:27.787267 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.787197    2574 generic.go:356] "Generic (PLEG): container finished" podID="87207e93-7303-4c9a-bbd9-cd7107e296c3" containerID="a42f17e72f002e44f4d7c884a86829e9c62fd79d0137bbfa82a2d17f957e26da" exitCode=0
Sep 11 22:05:27.787267 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.787246    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"
Sep 11 22:05:27.787465 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.787283    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv" event={"ID":"87207e93-7303-4c9a-bbd9-cd7107e296c3","Type":"ContainerDied","Data":"a42f17e72f002e44f4d7c884a86829e9c62fd79d0137bbfa82a2d17f957e26da"}
Sep 11 22:05:27.787465 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.787360    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv" event={"ID":"87207e93-7303-4c9a-bbd9-cd7107e296c3","Type":"ContainerDied","Data":"8b78803820bd292a4de71a1c655d39d298a277c3b9069437dade01286b356da9"}
Sep 11 22:05:27.787465 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.787394    2574 scope.go:117] "RemoveContainer" containerID="a42f17e72f002e44f4d7c884a86829e9c62fd79d0137bbfa82a2d17f957e26da"
Sep 11 22:05:27.788762 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.788741    2574 generic.go:356] "Generic (PLEG): container finished" podID="d2ec7893-db2a-4420-8d5e-03b6322687a7" containerID="104f3d2d392dff7a3b9d76652a2bad80170f6a0a1398b83c2eb29e1559bb994a" exitCode=0
Sep 11 22:05:27.788869 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.788777    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7" event={"ID":"d2ec7893-db2a-4420-8d5e-03b6322687a7","Type":"ContainerDied","Data":"104f3d2d392dff7a3b9d76652a2bad80170f6a0a1398b83c2eb29e1559bb994a"}
Sep 11 22:05:27.788869 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.788793    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"
Sep 11 22:05:27.788869 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.788816    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7" event={"ID":"d2ec7893-db2a-4420-8d5e-03b6322687a7","Type":"ContainerDied","Data":"91633d46cdc827219da4a03be5c990298a34764723fc303aed97e7382d583c21"}
Sep 11 22:05:27.798688 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.798668    2574 scope.go:117] "RemoveContainer" containerID="a42f17e72f002e44f4d7c884a86829e9c62fd79d0137bbfa82a2d17f957e26da"
Sep 11 22:05:27.798934 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:05:27.798911    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42f17e72f002e44f4d7c884a86829e9c62fd79d0137bbfa82a2d17f957e26da\": container with ID starting with a42f17e72f002e44f4d7c884a86829e9c62fd79d0137bbfa82a2d17f957e26da not found: ID does not exist" containerID="a42f17e72f002e44f4d7c884a86829e9c62fd79d0137bbfa82a2d17f957e26da"
Sep 11 22:05:27.799044 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.798938    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42f17e72f002e44f4d7c884a86829e9c62fd79d0137bbfa82a2d17f957e26da"} err="failed to get container status \"a42f17e72f002e44f4d7c884a86829e9c62fd79d0137bbfa82a2d17f957e26da\": rpc error: code = NotFound desc = could not find container \"a42f17e72f002e44f4d7c884a86829e9c62fd79d0137bbfa82a2d17f957e26da\": container with ID starting with a42f17e72f002e44f4d7c884a86829e9c62fd79d0137bbfa82a2d17f957e26da not found: ID does not exist"
Sep 11 22:05:27.799044 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.798954    2574 scope.go:117] "RemoveContainer" containerID="104f3d2d392dff7a3b9d76652a2bad80170f6a0a1398b83c2eb29e1559bb994a"
Sep 11 22:05:27.807451 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.807436    2574 scope.go:117] "RemoveContainer" containerID="104f3d2d392dff7a3b9d76652a2bad80170f6a0a1398b83c2eb29e1559bb994a"
Sep 11 22:05:27.807726 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:05:27.807707    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104f3d2d392dff7a3b9d76652a2bad80170f6a0a1398b83c2eb29e1559bb994a\": container with ID starting with 104f3d2d392dff7a3b9d76652a2bad80170f6a0a1398b83c2eb29e1559bb994a not found: ID does not exist" containerID="104f3d2d392dff7a3b9d76652a2bad80170f6a0a1398b83c2eb29e1559bb994a"
Sep 11 22:05:27.807803 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.807732    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104f3d2d392dff7a3b9d76652a2bad80170f6a0a1398b83c2eb29e1559bb994a"} err="failed to get container status \"104f3d2d392dff7a3b9d76652a2bad80170f6a0a1398b83c2eb29e1559bb994a\": rpc error: code = NotFound desc = could not find container \"104f3d2d392dff7a3b9d76652a2bad80170f6a0a1398b83c2eb29e1559bb994a\": container with ID starting with 104f3d2d392dff7a3b9d76652a2bad80170f6a0a1398b83c2eb29e1559bb994a not found: ID does not exist"
Sep 11 22:05:27.810303 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.810272    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:27.811520 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.811498    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"]
Sep 11 22:05:27.814583 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.814563    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0cd94651-66c6c66b46-tqkmv"]
Sep 11 22:05:27.827304 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.827285    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"]
Sep 11 22:05:27.830777 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:27.830759    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0cd94651-9f7477bf4-szxf7"]
Sep 11 22:05:28.075438 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:28.075366    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:28.424847 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:28.424809    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:28.525374 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:28.525345    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:29.209557 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:29.209521    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:29.720964 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:29.720923    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87207e93-7303-4c9a-bbd9-cd7107e296c3" path="/var/lib/kubelet/pods/87207e93-7303-4c9a-bbd9-cd7107e296c3/volumes"
Sep 11 22:05:29.721404 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:29.721384    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2ec7893-db2a-4420-8d5e-03b6322687a7" path="/var/lib/kubelet/pods/d2ec7893-db2a-4420-8d5e-03b6322687a7/volumes"
Sep 11 22:05:29.996857 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:29.996769    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:30.646201 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:30.646147    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:30.740594 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:30.740562    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:30.837904 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:30.837870    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:31.725649 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:31.725622    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:32.577976 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.577942    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"]
Sep 11 22:05:32.578378 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.578365    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ec7893-db2a-4420-8d5e-03b6322687a7" containerName="payload-pre-processing"
Sep 11 22:05:32.578429 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.578379    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ec7893-db2a-4420-8d5e-03b6322687a7" containerName="payload-pre-processing"
Sep 11 22:05:32.578429 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.578390    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c81ecc2a-4334-4918-8a87-0771282c99c8" containerName="revoke-keys"
Sep 11 22:05:32.578429 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.578395    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c81ecc2a-4334-4918-8a87-0771282c99c8" containerName="revoke-keys"
Sep 11 22:05:32.578429 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.578406    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87207e93-7303-4c9a-bbd9-cd7107e296c3" containerName="payload-processing"
Sep 11 22:05:32.578429 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.578411    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="87207e93-7303-4c9a-bbd9-cd7107e296c3" containerName="payload-processing"
Sep 11 22:05:32.578600 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.578487    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ec7893-db2a-4420-8d5e-03b6322687a7" containerName="payload-pre-processing"
Sep 11 22:05:32.578600 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.578494    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="87207e93-7303-4c9a-bbd9-cd7107e296c3" containerName="payload-processing"
Sep 11 22:05:32.578600 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.578502    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c81ecc2a-4334-4918-8a87-0771282c99c8" containerName="revoke-keys"
Sep 11 22:05:32.583201 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.583177    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:32.587481 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.587458    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6f399ea1-serving-cert\""
Sep 11 22:05:32.587627 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.587603    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 22:05:32.587704 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.587678    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:05:32.587763 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.587738    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 22:05:32.587906 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.587887    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mc9f9\""
Sep 11 22:05:32.595248 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.595183    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"]
Sep 11 22:05:32.614073 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.614037    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:32.669223 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.669187    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8137ab64-f54f-40cf-bd56-0453a63acad8-maas-api-tls\") pod \"maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:32.669390 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.669276    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8137ab64-f54f-40cf-bd56-0453a63acad8-openshift-service-ca\") pod \"maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:32.669390 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.669379    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8137ab64-f54f-40cf-bd56-0453a63acad8-metrics-certs\") pod \"maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:32.669492 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.669433    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4mb\" (UniqueName: \"kubernetes.io/projected/8137ab64-f54f-40cf-bd56-0453a63acad8-kube-api-access-gb4mb\") pod \"maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:32.684678 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.684650    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:32.770253 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.770194    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4mb\" (UniqueName: \"kubernetes.io/projected/8137ab64-f54f-40cf-bd56-0453a63acad8-kube-api-access-gb4mb\") pod \"maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:32.770787 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.770265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8137ab64-f54f-40cf-bd56-0453a63acad8-maas-api-tls\") pod \"maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:32.770787 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.770393    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8137ab64-f54f-40cf-bd56-0453a63acad8-openshift-service-ca\") pod \"maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:32.770787 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.770462    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8137ab64-f54f-40cf-bd56-0453a63acad8-metrics-certs\") pod \"maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:32.771185 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.771136    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8137ab64-f54f-40cf-bd56-0453a63acad8-openshift-service-ca\") pod \"maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:32.772736 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.772715    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8137ab64-f54f-40cf-bd56-0453a63acad8-maas-api-tls\") pod \"maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:32.772818 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.772778    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8137ab64-f54f-40cf-bd56-0453a63acad8-metrics-certs\") pod \"maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:32.779551 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.779532    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4mb\" (UniqueName: \"kubernetes.io/projected/8137ab64-f54f-40cf-bd56-0453a63acad8-kube-api-access-gb4mb\") pod \"maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:32.786401 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.786370    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:32.897417 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:32.897383    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:33.147395 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.147353    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:33.179261 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.179237    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"]
Sep 11 22:05:33.182159 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:05:33.182131    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8137ab64_f54f_40cf_bd56_0453a63acad8.slice/crio-d11cabba1698bd643a3a4e79a14e82e9dd83897366d1bad48840600e9657eea4 WatchSource:0}: Error finding container d11cabba1698bd643a3a4e79a14e82e9dd83897366d1bad48840600e9657eea4: Status 404 returned error can't find the container with id d11cabba1698bd643a3a4e79a14e82e9dd83897366d1bad48840600e9657eea4
Sep 11 22:05:33.244377 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.244345    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:33.253066 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.253042    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"]
Sep 11 22:05:33.257980 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.257964    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"
Sep 11 22:05:33.260774 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.260749    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6f399ea1\""
Sep 11 22:05:33.260874 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.260831    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6f399ea1-dockercfg-8ggdd\""
Sep 11 22:05:33.265836 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.265659    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"]
Sep 11 22:05:33.286698 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.286664    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"]
Sep 11 22:05:33.290986 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.290966    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"
Sep 11 22:05:33.299798 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.299779    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"]
Sep 11 22:05:33.375668 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.375630    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl877\" (UniqueName: \"kubernetes.io/projected/95d66e87-2038-42d1-a565-25b2b3c38ba6-kube-api-access-xl877\") pod \"payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb\" (UID: \"95d66e87-2038-42d1-a565-25b2b3c38ba6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"
Sep 11 22:05:33.375845 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.375766    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d66e87-2038-42d1-a565-25b2b3c38ba6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb\" (UID: \"95d66e87-2038-42d1-a565-25b2b3c38ba6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"
Sep 11 22:05:33.477426 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.477148    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl877\" (UniqueName: \"kubernetes.io/projected/95d66e87-2038-42d1-a565-25b2b3c38ba6-kube-api-access-xl877\") pod \"payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb\" (UID: \"95d66e87-2038-42d1-a565-25b2b3c38ba6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"
Sep 11 22:05:33.477586 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.477565    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfbrj\" (UniqueName: \"kubernetes.io/projected/bcaf84d0-c5ec-44d8-8a70-10e36322138f-kube-api-access-pfbrj\") pod \"payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8\" (UID: \"bcaf84d0-c5ec-44d8-8a70-10e36322138f\") " pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"
Sep 11 22:05:33.478135 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.477644    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcaf84d0-c5ec-44d8-8a70-10e36322138f-plugins-config-volume\") pod \"payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8\" (UID: \"bcaf84d0-c5ec-44d8-8a70-10e36322138f\") " pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"
Sep 11 22:05:33.478135 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.478040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d66e87-2038-42d1-a565-25b2b3c38ba6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb\" (UID: \"95d66e87-2038-42d1-a565-25b2b3c38ba6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"
Sep 11 22:05:33.482327 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.478984    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d66e87-2038-42d1-a565-25b2b3c38ba6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb\" (UID: \"95d66e87-2038-42d1-a565-25b2b3c38ba6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"
Sep 11 22:05:33.488122 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.488097    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl877\" (UniqueName: \"kubernetes.io/projected/95d66e87-2038-42d1-a565-25b2b3c38ba6-kube-api-access-xl877\") pod \"payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb\" (UID: \"95d66e87-2038-42d1-a565-25b2b3c38ba6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"
Sep 11 22:05:33.530554 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.530525    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:33.569659 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.569636    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"
Sep 11 22:05:33.578869 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.578850    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfbrj\" (UniqueName: \"kubernetes.io/projected/bcaf84d0-c5ec-44d8-8a70-10e36322138f-kube-api-access-pfbrj\") pod \"payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8\" (UID: \"bcaf84d0-c5ec-44d8-8a70-10e36322138f\") " pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"
Sep 11 22:05:33.578954 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.578883    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcaf84d0-c5ec-44d8-8a70-10e36322138f-plugins-config-volume\") pod \"payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8\" (UID: \"bcaf84d0-c5ec-44d8-8a70-10e36322138f\") " pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"
Sep 11 22:05:33.579497 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.579478    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcaf84d0-c5ec-44d8-8a70-10e36322138f-plugins-config-volume\") pod \"payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8\" (UID: \"bcaf84d0-c5ec-44d8-8a70-10e36322138f\") " pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"
Sep 11 22:05:33.586887 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.586868    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfbrj\" (UniqueName: \"kubernetes.io/projected/bcaf84d0-c5ec-44d8-8a70-10e36322138f-kube-api-access-pfbrj\") pod \"payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8\" (UID: \"bcaf84d0-c5ec-44d8-8a70-10e36322138f\") " pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"
Sep 11 22:05:33.602495 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.602468    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"
Sep 11 22:05:33.826637 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.826560    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k" event={"ID":"8137ab64-f54f-40cf-bd56-0453a63acad8","Type":"ContainerStarted","Data":"d11cabba1698bd643a3a4e79a14e82e9dd83897366d1bad48840600e9657eea4"}
Sep 11 22:05:33.847965 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.847939    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"]
Sep 11 22:05:33.849977 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:05:33.849948    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d66e87_2038_42d1_a565_25b2b3c38ba6.slice/crio-a90af3787df2d0028e81d9549852da3c0af38f5dc726a64ddfd316a652c84821 WatchSource:0}: Error finding container a90af3787df2d0028e81d9549852da3c0af38f5dc726a64ddfd316a652c84821: Status 404 returned error can't find the container with id a90af3787df2d0028e81d9549852da3c0af38f5dc726a64ddfd316a652c84821
Sep 11 22:05:33.895438 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:33.895415    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"]
Sep 11 22:05:33.900742 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:05:33.899799    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcaf84d0_c5ec_44d8_8a70_10e36322138f.slice/crio-71a20a321ff825366fed488313caf7eff01f3c730280c0d4ce495e28aa3de6ab WatchSource:0}: Error finding container 71a20a321ff825366fed488313caf7eff01f3c730280c0d4ce495e28aa3de6ab: Status 404 returned error can't find the container with id 71a20a321ff825366fed488313caf7eff01f3c730280c0d4ce495e28aa3de6ab
Sep 11 22:05:34.344948 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:34.344911    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:34.835193 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:34.835106    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8" event={"ID":"bcaf84d0-c5ec-44d8-8a70-10e36322138f","Type":"ContainerStarted","Data":"c761a0587355f2a77c5d25fadaec657cbb12bf15b96842d094e4c2805e4f1c95"}
Sep 11 22:05:34.835193 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:34.835175    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8" event={"ID":"bcaf84d0-c5ec-44d8-8a70-10e36322138f","Type":"ContainerStarted","Data":"71a20a321ff825366fed488313caf7eff01f3c730280c0d4ce495e28aa3de6ab"}
Sep 11 22:05:34.835728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:34.835290    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"
Sep 11 22:05:34.837040 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:34.837012    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb" event={"ID":"95d66e87-2038-42d1-a565-25b2b3c38ba6","Type":"ContainerStarted","Data":"425a9cb9e1b8fda58991f3a633c6e04921e795f52f5573311f36f98c3ec96455"}
Sep 11 22:05:34.837184 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:34.837060    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb" event={"ID":"95d66e87-2038-42d1-a565-25b2b3c38ba6","Type":"ContainerStarted","Data":"a90af3787df2d0028e81d9549852da3c0af38f5dc726a64ddfd316a652c84821"}
Sep 11 22:05:34.837184 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:34.837101    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"
Sep 11 22:05:34.855464 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:34.855296    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8" podStartSLOduration=1.855282388 podStartE2EDuration="1.855282388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:05:34.854678821 +0000 UTC m=+801.589790168" watchObservedRunningTime="2026-09-11 22:05:34.855282388 +0000 UTC m=+801.590393739"
Sep 11 22:05:34.873643 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:34.873601    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb" podStartSLOduration=1.873587878 podStartE2EDuration="1.873587878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:05:34.871011676 +0000 UTC m=+801.606123027" watchObservedRunningTime="2026-09-11 22:05:34.873587878 +0000 UTC m=+801.608699223"
Sep 11 22:05:35.122445 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:35.122228    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:35.850811 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:35.850778    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:36.849846 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:36.849804    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k" event={"ID":"8137ab64-f54f-40cf-bd56-0453a63acad8","Type":"ContainerStarted","Data":"f53ce627ac0e798295e66c964989c114a521d165b62b212d492f73295c0696b1"}
Sep 11 22:05:36.850034 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:36.849976    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:36.868461 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:36.868406    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k" podStartSLOduration=2.217679761 podStartE2EDuration="4.868390528s" totalImagesPullingTime="2.650710767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:05:36.866790701 +0000 UTC m=+803.601902050" watchObservedRunningTime="2026-09-11 22:05:36.868390528 +0000 UTC m=+803.603501873"
Sep 11 22:05:42.861177 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:42.861151    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:05:45.844683 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:45.844656    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"
Sep 11 22:05:45.845124 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:45.844956    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"
Sep 11 22:05:49.393651 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:49.393617    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:54.707746 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:54.707711    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:55.468211 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:55.468174    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:56.251612 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:56.251580    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:57.133108 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:57.133071    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:57.922700 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:57.922662    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:58.705546 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:58.705511    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:59.548565 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:05:59.548532    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:00.460169 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:00.460131    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:01.411197 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:01.411166    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:02.168067 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:02.168031    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:03.686271 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:03.686238    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:12.000208 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:12.000171    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:12.754038 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:12.753999    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"]
Sep 11 22:06:12.754373 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:12.754340    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k" podUID="8137ab64-f54f-40cf-bd56-0453a63acad8" containerName="maas-api" containerID="cri-o://f53ce627ac0e798295e66c964989c114a521d165b62b212d492f73295c0696b1" gracePeriod=30
Sep 11 22:06:12.800074 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:12.800047    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:12.831289 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:12.831262    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"]
Sep 11 22:06:12.831859 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:12.831830    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8" podUID="bcaf84d0-c5ec-44d8-8a70-10e36322138f" containerName="payload-processing" containerID="cri-o://c761a0587355f2a77c5d25fadaec657cbb12bf15b96842d094e4c2805e4f1c95" gracePeriod=30
Sep 11 22:06:12.837918 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:12.837894    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"]
Sep 11 22:06:12.838122 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:12.838096    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb" podUID="95d66e87-2038-42d1-a565-25b2b3c38ba6" containerName="payload-pre-processing" containerID="cri-o://425a9cb9e1b8fda58991f3a633c6e04921e795f52f5573311f36f98c3ec96455" gracePeriod=30
Sep 11 22:06:12.855796 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:12.855753    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k" podUID="8137ab64-f54f-40cf-bd56-0453a63acad8" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.50:8443/health\": dial tcp 10.132.0.50:8443: connect: connection refused"
Sep 11 22:06:12.881006 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:12.880930    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:12.976808 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:12.976775    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:13.048803 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.048774    2574 generic.go:356] "Generic (PLEG): container finished" podID="95d66e87-2038-42d1-a565-25b2b3c38ba6" containerID="425a9cb9e1b8fda58991f3a633c6e04921e795f52f5573311f36f98c3ec96455" exitCode=0
Sep 11 22:06:13.049224 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.048850    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb" event={"ID":"95d66e87-2038-42d1-a565-25b2b3c38ba6","Type":"ContainerDied","Data":"425a9cb9e1b8fda58991f3a633c6e04921e795f52f5573311f36f98c3ec96455"}
Sep 11 22:06:13.052330 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.051426    2574 generic.go:356] "Generic (PLEG): container finished" podID="bcaf84d0-c5ec-44d8-8a70-10e36322138f" containerID="c761a0587355f2a77c5d25fadaec657cbb12bf15b96842d094e4c2805e4f1c95" exitCode=0
Sep 11 22:06:13.052330 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.051561    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8" event={"ID":"bcaf84d0-c5ec-44d8-8a70-10e36322138f","Type":"ContainerDied","Data":"c761a0587355f2a77c5d25fadaec657cbb12bf15b96842d094e4c2805e4f1c95"}
Sep 11 22:06:13.053764 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.053740    2574 generic.go:356] "Generic (PLEG): container finished" podID="8137ab64-f54f-40cf-bd56-0453a63acad8" containerID="f53ce627ac0e798295e66c964989c114a521d165b62b212d492f73295c0696b1" exitCode=0
Sep 11 22:06:13.053885 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.053838    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k" event={"ID":"8137ab64-f54f-40cf-bd56-0453a63acad8","Type":"ContainerDied","Data":"f53ce627ac0e798295e66c964989c114a521d165b62b212d492f73295c0696b1"}
Sep 11 22:06:13.102921 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.102890    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"
Sep 11 22:06:13.109078 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.109033    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:13.134166 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.134142    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:06:13.153382 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.153361    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"
Sep 11 22:06:13.230599 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.230560    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:13.242347 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.242324    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d66e87-2038-42d1-a565-25b2b3c38ba6-plugins-config-volume\") pod \"95d66e87-2038-42d1-a565-25b2b3c38ba6\" (UID: \"95d66e87-2038-42d1-a565-25b2b3c38ba6\") "
Sep 11 22:06:13.242504 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.242375    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcaf84d0-c5ec-44d8-8a70-10e36322138f-plugins-config-volume\") pod \"bcaf84d0-c5ec-44d8-8a70-10e36322138f\" (UID: \"bcaf84d0-c5ec-44d8-8a70-10e36322138f\") "
Sep 11 22:06:13.242504 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.242411    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl877\" (UniqueName: \"kubernetes.io/projected/95d66e87-2038-42d1-a565-25b2b3c38ba6-kube-api-access-xl877\") pod \"95d66e87-2038-42d1-a565-25b2b3c38ba6\" (UID: \"95d66e87-2038-42d1-a565-25b2b3c38ba6\") "
Sep 11 22:06:13.242504 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.242487    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8137ab64-f54f-40cf-bd56-0453a63acad8-metrics-certs\") pod \"8137ab64-f54f-40cf-bd56-0453a63acad8\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") "
Sep 11 22:06:13.242668 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.242516    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfbrj\" (UniqueName: \"kubernetes.io/projected/bcaf84d0-c5ec-44d8-8a70-10e36322138f-kube-api-access-pfbrj\") pod \"bcaf84d0-c5ec-44d8-8a70-10e36322138f\" (UID: \"bcaf84d0-c5ec-44d8-8a70-10e36322138f\") "
Sep 11 22:06:13.242668 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.242550    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb4mb\" (UniqueName: \"kubernetes.io/projected/8137ab64-f54f-40cf-bd56-0453a63acad8-kube-api-access-gb4mb\") pod \"8137ab64-f54f-40cf-bd56-0453a63acad8\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") "
Sep 11 22:06:13.242668 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.242632    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8137ab64-f54f-40cf-bd56-0453a63acad8-maas-api-tls\") pod \"8137ab64-f54f-40cf-bd56-0453a63acad8\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") "
Sep 11 22:06:13.242668 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.242663    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8137ab64-f54f-40cf-bd56-0453a63acad8-openshift-service-ca\") pod \"8137ab64-f54f-40cf-bd56-0453a63acad8\" (UID: \"8137ab64-f54f-40cf-bd56-0453a63acad8\") "
Sep 11 22:06:13.242850 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.242681    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d66e87-2038-42d1-a565-25b2b3c38ba6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95d66e87-2038-42d1-a565-25b2b3c38ba6" (UID: "95d66e87-2038-42d1-a565-25b2b3c38ba6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:06:13.242850 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.242719    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcaf84d0-c5ec-44d8-8a70-10e36322138f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bcaf84d0-c5ec-44d8-8a70-10e36322138f" (UID: "bcaf84d0-c5ec-44d8-8a70-10e36322138f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:06:13.242998 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.242981    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d66e87-2038-42d1-a565-25b2b3c38ba6-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:06:13.243057 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.243003    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcaf84d0-c5ec-44d8-8a70-10e36322138f-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:06:13.243249 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.243222    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8137ab64-f54f-40cf-bd56-0453a63acad8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8137ab64-f54f-40cf-bd56-0453a63acad8" (UID: "8137ab64-f54f-40cf-bd56-0453a63acad8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:06:13.244958 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.244926    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8137ab64-f54f-40cf-bd56-0453a63acad8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8137ab64-f54f-40cf-bd56-0453a63acad8" (UID: "8137ab64-f54f-40cf-bd56-0453a63acad8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:06:13.245087 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.245050    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8137ab64-f54f-40cf-bd56-0453a63acad8-kube-api-access-gb4mb" (OuterVolumeSpecName: "kube-api-access-gb4mb") pod "8137ab64-f54f-40cf-bd56-0453a63acad8" (UID: "8137ab64-f54f-40cf-bd56-0453a63acad8"). InnerVolumeSpecName "kube-api-access-gb4mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:06:13.245174 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.245106    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8137ab64-f54f-40cf-bd56-0453a63acad8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8137ab64-f54f-40cf-bd56-0453a63acad8" (UID: "8137ab64-f54f-40cf-bd56-0453a63acad8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:06:13.245174 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.245118    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcaf84d0-c5ec-44d8-8a70-10e36322138f-kube-api-access-pfbrj" (OuterVolumeSpecName: "kube-api-access-pfbrj") pod "bcaf84d0-c5ec-44d8-8a70-10e36322138f" (UID: "bcaf84d0-c5ec-44d8-8a70-10e36322138f"). InnerVolumeSpecName "kube-api-access-pfbrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:06:13.245325 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.245295    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d66e87-2038-42d1-a565-25b2b3c38ba6-kube-api-access-xl877" (OuterVolumeSpecName: "kube-api-access-xl877") pod "95d66e87-2038-42d1-a565-25b2b3c38ba6" (UID: "95d66e87-2038-42d1-a565-25b2b3c38ba6"). InnerVolumeSpecName "kube-api-access-xl877". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:06:13.344517 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.344486    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl877\" (UniqueName: \"kubernetes.io/projected/95d66e87-2038-42d1-a565-25b2b3c38ba6-kube-api-access-xl877\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:06:13.344517 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.344514    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8137ab64-f54f-40cf-bd56-0453a63acad8-metrics-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:06:13.344710 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.344527    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfbrj\" (UniqueName: \"kubernetes.io/projected/bcaf84d0-c5ec-44d8-8a70-10e36322138f-kube-api-access-pfbrj\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:06:13.344710 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.344540    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4mb\" (UniqueName: \"kubernetes.io/projected/8137ab64-f54f-40cf-bd56-0453a63acad8-kube-api-access-gb4mb\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:06:13.344710 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.344553    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8137ab64-f54f-40cf-bd56-0453a63acad8-maas-api-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:06:13.344710 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.344563    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8137ab64-f54f-40cf-bd56-0453a63acad8-openshift-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:06:13.527956 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:13.527916    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:14.061080 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.060995    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"
Sep 11 22:06:14.061080 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.061005    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k" event={"ID":"8137ab64-f54f-40cf-bd56-0453a63acad8","Type":"ContainerDied","Data":"d11cabba1698bd643a3a4e79a14e82e9dd83897366d1bad48840600e9657eea4"}
Sep 11 22:06:14.061579 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.061104    2574 scope.go:117] "RemoveContainer" containerID="f53ce627ac0e798295e66c964989c114a521d165b62b212d492f73295c0696b1"
Sep 11 22:06:14.063473 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.062991    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"
Sep 11 22:06:14.063473 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.063023    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb" event={"ID":"95d66e87-2038-42d1-a565-25b2b3c38ba6","Type":"ContainerDied","Data":"a90af3787df2d0028e81d9549852da3c0af38f5dc726a64ddfd316a652c84821"}
Sep 11 22:06:14.065717 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.065691    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8" event={"ID":"bcaf84d0-c5ec-44d8-8a70-10e36322138f","Type":"ContainerDied","Data":"71a20a321ff825366fed488313caf7eff01f3c730280c0d4ce495e28aa3de6ab"}
Sep 11 22:06:14.065829 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.065735    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"
Sep 11 22:06:14.072480 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.072459    2574 scope.go:117] "RemoveContainer" containerID="425a9cb9e1b8fda58991f3a633c6e04921e795f52f5573311f36f98c3ec96455"
Sep 11 22:06:14.080259 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.080241    2574 scope.go:117] "RemoveContainer" containerID="c761a0587355f2a77c5d25fadaec657cbb12bf15b96842d094e4c2805e4f1c95"
Sep 11 22:06:14.085444 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.085423    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"]
Sep 11 22:06:14.090712 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.090693    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f399ea1-7f6fdcd6c5-svl7k"]
Sep 11 22:06:14.105372 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.105350    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"]
Sep 11 22:06:14.108825 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.108801    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f399ea1-5b86d99bbb-n9fqb"]
Sep 11 22:06:14.120483 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.120463    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"]
Sep 11 22:06:14.125928 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.125908    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f399ea1-6b8d985759-wkmj8"]
Sep 11 22:06:14.288086 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.288039    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:14.418158 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.418125    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:14.745367 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:14.745267    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:15.075494 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:15.075409    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:15.283344 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:15.283292    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:15.721863 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:15.721821    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8137ab64-f54f-40cf-bd56-0453a63acad8" path="/var/lib/kubelet/pods/8137ab64-f54f-40cf-bd56-0453a63acad8/volumes"
Sep 11 22:06:15.722402 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:15.722382    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95d66e87-2038-42d1-a565-25b2b3c38ba6" path="/var/lib/kubelet/pods/95d66e87-2038-42d1-a565-25b2b3c38ba6/volumes"
Sep 11 22:06:15.722980 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:15.722962    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcaf84d0-c5ec-44d8-8a70-10e36322138f" path="/var/lib/kubelet/pods/bcaf84d0-c5ec-44d8-8a70-10e36322138f/volumes"
Sep 11 22:06:15.877517 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:15.877485    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:16.090332 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:16.090293    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:16.239485 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:16.239447    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:17.223155 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:17.223119    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:17.942451 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:17.942411    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:18.036867 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:18.036834    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:18.205139 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:18.205069    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:19.142067 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:19.142029    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:19.826170 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:19.826134    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:20.504003 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:20.503971    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:21.307515 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:21.307480    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:22.195414 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.195381    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:22.824136 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.824106    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"]
Sep 11 22:06:22.824828 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.824804    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d66e87-2038-42d1-a565-25b2b3c38ba6" containerName="payload-pre-processing"
Sep 11 22:06:22.825002 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.824975    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d66e87-2038-42d1-a565-25b2b3c38ba6" containerName="payload-pre-processing"
Sep 11 22:06:22.825121 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.825039    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8137ab64-f54f-40cf-bd56-0453a63acad8" containerName="maas-api"
Sep 11 22:06:22.825121 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.825046    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8137ab64-f54f-40cf-bd56-0453a63acad8" containerName="maas-api"
Sep 11 22:06:22.825121 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.825056    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcaf84d0-c5ec-44d8-8a70-10e36322138f" containerName="payload-processing"
Sep 11 22:06:22.825121 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.825064    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcaf84d0-c5ec-44d8-8a70-10e36322138f" containerName="payload-processing"
Sep 11 22:06:22.825359 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.825155    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8137ab64-f54f-40cf-bd56-0453a63acad8" containerName="maas-api"
Sep 11 22:06:22.825359 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.825168    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="95d66e87-2038-42d1-a565-25b2b3c38ba6" containerName="payload-pre-processing"
Sep 11 22:06:22.825359 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.825176    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcaf84d0-c5ec-44d8-8a70-10e36322138f" containerName="payload-processing"
Sep 11 22:06:22.830251 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.830228    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:22.833151 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.833126    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2c9db036-serving-cert\""
Sep 11 22:06:22.833326 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.833284    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 22:06:22.833326 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.833160    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 22:06:22.833464 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.833290    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mc9f9\""
Sep 11 22:06:22.833464 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.833346    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:06:22.845042 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.845002    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"]
Sep 11 22:06:22.873478 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.873451    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:22.931671 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.931641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-openshift-service-ca\") pod \"maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:22.931824 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.931721    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hklg7\" (UniqueName: \"kubernetes.io/projected/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-kube-api-access-hklg7\") pod \"maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:22.931824 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.931759    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-metrics-certs\") pod \"maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:22.931824 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.931788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-maas-api-tls\") pod \"maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:22.975460 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:22.975426    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:23.033171 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.033139    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-openshift-service-ca\") pod \"maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:23.033347 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.033221    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hklg7\" (UniqueName: \"kubernetes.io/projected/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-kube-api-access-hklg7\") pod \"maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:23.033347 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.033263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-metrics-certs\") pod \"maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:23.033347 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.033318    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-maas-api-tls\") pod \"maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:23.033477 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:06:23.033422    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-serving-cert: secret "maas-api-e2e-derive-2c9db036-serving-cert" not found
Sep 11 22:06:23.033519 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:06:23.033506    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-maas-api-tls podName:4bd34b82-d7d0-4e92-839f-6e6a92a6afd3 nodeName:}" failed. No retries permitted until 2026-09-11 22:06:23.533485449 +0000 UTC m=+850.268596779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-maas-api-tls") pod "maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq" (UID: "4bd34b82-d7d0-4e92-839f-6e6a92a6afd3") : secret "maas-api-e2e-derive-2c9db036-serving-cert" not found
Sep 11 22:06:23.033925 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.033905    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-openshift-service-ca\") pod \"maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:23.035838 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.035815    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-metrics-certs\") pod \"maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:23.041761 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.041739    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hklg7\" (UniqueName: \"kubernetes.io/projected/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-kube-api-access-hklg7\") pod \"maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:23.095291 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.095267    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:23.424633 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.424547    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:23.525190 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.525151    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:23.526213 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.526189    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"]
Sep 11 22:06:23.529743 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.529727    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"
Sep 11 22:06:23.532662 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.532643    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2c9db036\""
Sep 11 22:06:23.532746 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.532678    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2c9db036-dockercfg-5tnxm\""
Sep 11 22:06:23.537572 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.537547    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-maas-api-tls\") pod \"maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:23.537677 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.537597    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4826bcc3-393b-4af4-ad7a-7a78ad11d838-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh\" (UID: \"4826bcc3-393b-4af4-ad7a-7a78ad11d838\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"
Sep 11 22:06:23.537794 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.537768    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9f5f\" (UniqueName: \"kubernetes.io/projected/4826bcc3-393b-4af4-ad7a-7a78ad11d838-kube-api-access-d9f5f\") pod \"payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh\" (UID: \"4826bcc3-393b-4af4-ad7a-7a78ad11d838\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"
Sep 11 22:06:23.539574 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.539550    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"]
Sep 11 22:06:23.540185 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.540166    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-maas-api-tls\") pod \"maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:23.561663 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.561631    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"]
Sep 11 22:06:23.565154 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.565137    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"
Sep 11 22:06:23.575654 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.575630    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"]
Sep 11 22:06:23.638884 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.638851    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a-plugins-config-volume\") pod \"payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg\" (UID: \"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a\") " pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"
Sep 11 22:06:23.639042 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.638895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4826bcc3-393b-4af4-ad7a-7a78ad11d838-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh\" (UID: \"4826bcc3-393b-4af4-ad7a-7a78ad11d838\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"
Sep 11 22:06:23.639042 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.638986    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl6f2\" (UniqueName: \"kubernetes.io/projected/a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a-kube-api-access-jl6f2\") pod \"payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg\" (UID: \"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a\") " pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"
Sep 11 22:06:23.639042 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.639021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9f5f\" (UniqueName: \"kubernetes.io/projected/4826bcc3-393b-4af4-ad7a-7a78ad11d838-kube-api-access-d9f5f\") pod \"payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh\" (UID: \"4826bcc3-393b-4af4-ad7a-7a78ad11d838\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"
Sep 11 22:06:23.639761 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.639737    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4826bcc3-393b-4af4-ad7a-7a78ad11d838-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh\" (UID: \"4826bcc3-393b-4af4-ad7a-7a78ad11d838\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"
Sep 11 22:06:23.646979 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.646959    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9f5f\" (UniqueName: \"kubernetes.io/projected/4826bcc3-393b-4af4-ad7a-7a78ad11d838-kube-api-access-d9f5f\") pod \"payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh\" (UID: \"4826bcc3-393b-4af4-ad7a-7a78ad11d838\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"
Sep 11 22:06:23.739836 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.739757    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl6f2\" (UniqueName: \"kubernetes.io/projected/a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a-kube-api-access-jl6f2\") pod \"payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg\" (UID: \"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a\") " pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"
Sep 11 22:06:23.739993 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.739828    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a-plugins-config-volume\") pod \"payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg\" (UID: \"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a\") " pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"
Sep 11 22:06:23.740579 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.740554    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a-plugins-config-volume\") pod \"payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg\" (UID: \"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a\") " pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"
Sep 11 22:06:23.741680 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.741663    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:23.747651 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.747632    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl6f2\" (UniqueName: \"kubernetes.io/projected/a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a-kube-api-access-jl6f2\") pod \"payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg\" (UID: \"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a\") " pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"
Sep 11 22:06:23.841531 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.841501    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"
Sep 11 22:06:23.875796 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:23.875766    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"
Sep 11 22:06:24.017991 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:24.017964    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"]
Sep 11 22:06:24.020683 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:06:24.020653    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd34b82_d7d0_4e92_839f_6e6a92a6afd3.slice/crio-456315324c71569ec24a07e9cca008004362bfe898c5ea92239ddd2503841779 WatchSource:0}: Error finding container 456315324c71569ec24a07e9cca008004362bfe898c5ea92239ddd2503841779: Status 404 returned error can't find the container with id 456315324c71569ec24a07e9cca008004362bfe898c5ea92239ddd2503841779
Sep 11 22:06:24.113502 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:24.113476    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"]
Sep 11 22:06:24.115441 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:06:24.115409    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4826bcc3_393b_4af4_ad7a_7a78ad11d838.slice/crio-0faae05c786c8381257791181780aa0efe071989685456a3576a3862bdaf5a31 WatchSource:0}: Error finding container 0faae05c786c8381257791181780aa0efe071989685456a3576a3862bdaf5a31: Status 404 returned error can't find the container with id 0faae05c786c8381257791181780aa0efe071989685456a3576a3862bdaf5a31
Sep 11 22:06:24.121578 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:24.121555    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq" event={"ID":"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3","Type":"ContainerStarted","Data":"456315324c71569ec24a07e9cca008004362bfe898c5ea92239ddd2503841779"}
Sep 11 22:06:24.161206 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:24.161183    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"]
Sep 11 22:06:24.171324 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:06:24.171288    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67dd6ca_b6cd_47fc_93dc_3fdf41c7373a.slice/crio-aa67391516a47c759e26a449d1cdd6ece2138872459689de2f62f7036aafc4a7 WatchSource:0}: Error finding container aa67391516a47c759e26a449d1cdd6ece2138872459689de2f62f7036aafc4a7: Status 404 returned error can't find the container with id aa67391516a47c759e26a449d1cdd6ece2138872459689de2f62f7036aafc4a7
Sep 11 22:06:25.129534 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:25.129500    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg" event={"ID":"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a","Type":"ContainerStarted","Data":"43bb90a6669128459b49902d0f30ac0997c90d775937fb0aab15e61e2fdc2a01"}
Sep 11 22:06:25.130010 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:25.129560    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg" event={"ID":"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a","Type":"ContainerStarted","Data":"aa67391516a47c759e26a449d1cdd6ece2138872459689de2f62f7036aafc4a7"}
Sep 11 22:06:25.130010 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:25.129682    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"
Sep 11 22:06:25.131383 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:25.131355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh" event={"ID":"4826bcc3-393b-4af4-ad7a-7a78ad11d838","Type":"ContainerStarted","Data":"68b5de5fdb2cb22e681f9c8abc8580de2eacce1d3f54e68418843c756891ef85"}
Sep 11 22:06:25.131533 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:25.131405    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh" event={"ID":"4826bcc3-393b-4af4-ad7a-7a78ad11d838","Type":"ContainerStarted","Data":"0faae05c786c8381257791181780aa0efe071989685456a3576a3862bdaf5a31"}
Sep 11 22:06:25.131533 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:25.131433    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"
Sep 11 22:06:25.133213 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:25.133189    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq" event={"ID":"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3","Type":"ContainerStarted","Data":"0cc412ba167c04a5a4dbc337adf7555df39693c91213debdbddfe357c51ae442"}
Sep 11 22:06:25.133351 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:25.133328    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:25.146617 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:25.146573    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg" podStartSLOduration=2.146556845 podStartE2EDuration="2.146556845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:06:25.144828581 +0000 UTC m=+851.879939931" watchObservedRunningTime="2026-09-11 22:06:25.146556845 +0000 UTC m=+851.881668191"
Sep 11 22:06:25.162902 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:25.162862    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh" podStartSLOduration=2.162850713 podStartE2EDuration="2.162850713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:06:25.161014965 +0000 UTC m=+851.896126311" watchObservedRunningTime="2026-09-11 22:06:25.162850713 +0000 UTC m=+851.897962055"
Sep 11 22:06:25.178882 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:25.178843    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq" podStartSLOduration=2.57814768 podStartE2EDuration="3.178830392s" totalImagesPullingTime="600.682712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:06:25.176731803 +0000 UTC m=+851.911843149" watchObservedRunningTime="2026-09-11 22:06:25.178830392 +0000 UTC m=+851.913941783"
Sep 11 22:06:27.379919 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:27.379878    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:31.143584 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:31.143554    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:06:36.140115 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:36.140081    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"
Sep 11 22:06:36.140581 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:36.140151    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"
Sep 11 22:06:39.513826 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:39.513775    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:44.690809 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:44.690778    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:46.267490 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:46.267458    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:47.568741 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:47.568705    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:48.274732 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:48.274703    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n"]
Sep 11 22:06:48.278199 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:48.278180    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n"
Sep 11 22:06:48.281614 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:48.281599    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:06:48.288562 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:48.288536    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n"]
Sep 11 22:06:48.368630 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:48.368590    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8c0e1-bd44-4116-8d00-2305b02a2740-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7d5ec8-s642n\" (UID: \"b6e8c0e1-bd44-4116-8d00-2305b02a2740\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n"
Sep 11 22:06:48.469428 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:48.469394    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8c0e1-bd44-4116-8d00-2305b02a2740-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7d5ec8-s642n\" (UID: \"b6e8c0e1-bd44-4116-8d00-2305b02a2740\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n"
Sep 11 22:06:48.469988 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:48.469968    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8c0e1-bd44-4116-8d00-2305b02a2740-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7d5ec8-s642n\" (UID: \"b6e8c0e1-bd44-4116-8d00-2305b02a2740\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n"
Sep 11 22:06:48.589844 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:48.589822    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n"
Sep 11 22:06:48.862415 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:48.862392    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n"]
Sep 11 22:06:48.863757 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:06:48.863727    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e8c0e1_bd44_4116_8d00_2305b02a2740.slice/crio-382137376fcb3f0e31dbf1c138d8d19c88e4a5a890b05f63a7f27177844402e1 WatchSource:0}: Error finding container 382137376fcb3f0e31dbf1c138d8d19c88e4a5a890b05f63a7f27177844402e1: Status 404 returned error can't find the container with id 382137376fcb3f0e31dbf1c138d8d19c88e4a5a890b05f63a7f27177844402e1
Sep 11 22:06:48.912341 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:48.912299    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:49.264898 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:49.264814    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n" event={"ID":"b6e8c0e1-bd44-4116-8d00-2305b02a2740","Type":"ContainerStarted","Data":"e63154adb21cebe49be7edc43793d10064386a018b65e7988cf62c117cd64e43"}
Sep 11 22:06:49.264898 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:49.264863    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n" event={"ID":"b6e8c0e1-bd44-4116-8d00-2305b02a2740","Type":"ContainerStarted","Data":"382137376fcb3f0e31dbf1c138d8d19c88e4a5a890b05f63a7f27177844402e1"}
Sep 11 22:06:49.298192 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:49.298150    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n" podStartSLOduration=1.298137104 podStartE2EDuration="1.298137104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:06:49.296139539 +0000 UTC m=+876.031250888" watchObservedRunningTime="2026-09-11 22:06:49.298137104 +0000 UTC m=+876.033248449"
Sep 11 22:06:49.890485 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:49.890456    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:50.272058 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:50.271966    2574 generic.go:356] "Generic (PLEG): container finished" podID="b6e8c0e1-bd44-4116-8d00-2305b02a2740" containerID="e63154adb21cebe49be7edc43793d10064386a018b65e7988cf62c117cd64e43" exitCode=0
Sep 11 22:06:50.272058 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:50.272010    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n" event={"ID":"b6e8c0e1-bd44-4116-8d00-2305b02a2740","Type":"ContainerDied","Data":"e63154adb21cebe49be7edc43793d10064386a018b65e7988cf62c117cd64e43"}
Sep 11 22:06:50.810845 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:50.810810    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:51.414963 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:51.414936    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n"
Sep 11 22:06:51.499919 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:51.499892    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8c0e1-bd44-4116-8d00-2305b02a2740-maas-api-service-ca\") pod \"b6e8c0e1-bd44-4116-8d00-2305b02a2740\" (UID: \"b6e8c0e1-bd44-4116-8d00-2305b02a2740\") "
Sep 11 22:06:51.500221 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:51.500202    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e8c0e1-bd44-4116-8d00-2305b02a2740-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b6e8c0e1-bd44-4116-8d00-2305b02a2740" (UID: "b6e8c0e1-bd44-4116-8d00-2305b02a2740"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:06:51.600687 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:51.600654    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8c0e1-bd44-4116-8d00-2305b02a2740-maas-api-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:06:51.718514 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:51.718468    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:52.283559 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:52.283528    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n"
Sep 11 22:06:52.283559 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:52.283546    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n" event={"ID":"b6e8c0e1-bd44-4116-8d00-2305b02a2740","Type":"ContainerDied","Data":"382137376fcb3f0e31dbf1c138d8d19c88e4a5a890b05f63a7f27177844402e1"}
Sep 11 22:06:52.283782 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:52.283596    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="382137376fcb3f0e31dbf1c138d8d19c88e4a5a890b05f63a7f27177844402e1"
Sep 11 22:06:52.663892 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:52.663861    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:53.384535 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:53.384498    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:54.327224 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:54.327184    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:55.106898 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:55.106854    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:58.407057 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:58.407021    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:59.019471 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.019433    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:59.054226 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.054192    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"]
Sep 11 22:06:59.054520 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.054488    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d" podUID="d5aed87d-88ba-46ff-93e2-d2a658f32b62" containerName="payload-processing" containerID="cri-o://1152859c2f36d2c326e152d170c7cac23dc3866a2123314ab30d753ad6cbf625" gracePeriod=30
Sep 11 22:06:59.070053 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.070022    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"]
Sep 11 22:06:59.070294 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.070268    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr" podUID="c2246aba-6a16-4848-9509-5fa42eb93d3b" containerName="payload-pre-processing" containerID="cri-o://581bd249bc5a69e882730d1783a40f48b4783f47fba33695d7f9657e1fa65a5a" gracePeriod=30
Sep 11 22:06:59.162042 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.162003    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:59.263241 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.263197    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:59.324498 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.324464    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n"]
Sep 11 22:06:59.326906 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.326864    2574 generic.go:356] "Generic (PLEG): container finished" podID="c2246aba-6a16-4848-9509-5fa42eb93d3b" containerID="581bd249bc5a69e882730d1783a40f48b4783f47fba33695d7f9657e1fa65a5a" exitCode=0
Sep 11 22:06:59.327114 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.327008    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr" event={"ID":"c2246aba-6a16-4848-9509-5fa42eb93d3b","Type":"ContainerDied","Data":"581bd249bc5a69e882730d1783a40f48b4783f47fba33695d7f9657e1fa65a5a"}
Sep 11 22:06:59.329100 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.329075    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7d5ec8-s642n"]
Sep 11 22:06:59.329720 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.329692    2574 generic.go:356] "Generic (PLEG): container finished" podID="d5aed87d-88ba-46ff-93e2-d2a658f32b62" containerID="1152859c2f36d2c326e152d170c7cac23dc3866a2123314ab30d753ad6cbf625" exitCode=0
Sep 11 22:06:59.329831 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.329720    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d" event={"ID":"d5aed87d-88ba-46ff-93e2-d2a658f32b62","Type":"ContainerDied","Data":"1152859c2f36d2c326e152d170c7cac23dc3866a2123314ab30d753ad6cbf625"}
Sep 11 22:06:59.330006 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.329984    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:59.359466 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.359449    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"
Sep 11 22:06:59.362455 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.362436    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"
Sep 11 22:06:59.473832 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.473800    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aed87d-88ba-46ff-93e2-d2a658f32b62-plugins-config-volume\") pod \"d5aed87d-88ba-46ff-93e2-d2a658f32b62\" (UID: \"d5aed87d-88ba-46ff-93e2-d2a658f32b62\") "
Sep 11 22:06:59.474209 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.473852    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbsj5\" (UniqueName: \"kubernetes.io/projected/c2246aba-6a16-4848-9509-5fa42eb93d3b-kube-api-access-rbsj5\") pod \"c2246aba-6a16-4848-9509-5fa42eb93d3b\" (UID: \"c2246aba-6a16-4848-9509-5fa42eb93d3b\") "
Sep 11 22:06:59.474209 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.473905    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wl9k\" (UniqueName: \"kubernetes.io/projected/d5aed87d-88ba-46ff-93e2-d2a658f32b62-kube-api-access-9wl9k\") pod \"d5aed87d-88ba-46ff-93e2-d2a658f32b62\" (UID: \"d5aed87d-88ba-46ff-93e2-d2a658f32b62\") "
Sep 11 22:06:59.474209 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.473962    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2246aba-6a16-4848-9509-5fa42eb93d3b-plugins-config-volume\") pod \"c2246aba-6a16-4848-9509-5fa42eb93d3b\" (UID: \"c2246aba-6a16-4848-9509-5fa42eb93d3b\") "
Sep 11 22:06:59.474209 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.474183    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5aed87d-88ba-46ff-93e2-d2a658f32b62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5aed87d-88ba-46ff-93e2-d2a658f32b62" (UID: "d5aed87d-88ba-46ff-93e2-d2a658f32b62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:06:59.474464 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.474430    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2246aba-6a16-4848-9509-5fa42eb93d3b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2246aba-6a16-4848-9509-5fa42eb93d3b" (UID: "c2246aba-6a16-4848-9509-5fa42eb93d3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:06:59.476058 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.476033    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5aed87d-88ba-46ff-93e2-d2a658f32b62-kube-api-access-9wl9k" (OuterVolumeSpecName: "kube-api-access-9wl9k") pod "d5aed87d-88ba-46ff-93e2-d2a658f32b62" (UID: "d5aed87d-88ba-46ff-93e2-d2a658f32b62"). InnerVolumeSpecName "kube-api-access-9wl9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:06:59.476058 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.476053    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2246aba-6a16-4848-9509-5fa42eb93d3b-kube-api-access-rbsj5" (OuterVolumeSpecName: "kube-api-access-rbsj5") pod "c2246aba-6a16-4848-9509-5fa42eb93d3b" (UID: "c2246aba-6a16-4848-9509-5fa42eb93d3b"). InnerVolumeSpecName "kube-api-access-rbsj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:06:59.563638 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.563559    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:59.575758 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.575731    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wl9k\" (UniqueName: \"kubernetes.io/projected/d5aed87d-88ba-46ff-93e2-d2a658f32b62-kube-api-access-9wl9k\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:06:59.575758 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.575753    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2246aba-6a16-4848-9509-5fa42eb93d3b-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:06:59.575920 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.575764    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aed87d-88ba-46ff-93e2-d2a658f32b62-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:06:59.575920 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.575773    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbsj5\" (UniqueName: \"kubernetes.io/projected/c2246aba-6a16-4848-9509-5fa42eb93d3b-kube-api-access-rbsj5\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:06:59.691668 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.691633    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:59.720909 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:06:59.720880    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6e8c0e1-bd44-4116-8d00-2305b02a2740" path="/var/lib/kubelet/pods/b6e8c0e1-bd44-4116-8d00-2305b02a2740/volumes"
Sep 11 22:07:00.305254 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:00.305215    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:00.335944 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:00.335914    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"
Sep 11 22:07:00.336109 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:00.335945    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr" event={"ID":"c2246aba-6a16-4848-9509-5fa42eb93d3b","Type":"ContainerDied","Data":"904a2a932dd5c565f3a5950764894b45b6a30c25c43ac5448b83c335a2d63682"}
Sep 11 22:07:00.336109 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:00.336012    2574 scope.go:117] "RemoveContainer" containerID="581bd249bc5a69e882730d1783a40f48b4783f47fba33695d7f9657e1fa65a5a"
Sep 11 22:07:00.337450 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:00.337359    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d" event={"ID":"d5aed87d-88ba-46ff-93e2-d2a658f32b62","Type":"ContainerDied","Data":"2053ca28854de5591fefeab1546a10a315f34d998945808b805bc784c8511b6f"}
Sep 11 22:07:00.337450 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:00.337431    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"
Sep 11 22:07:00.345097 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:00.345080    2574 scope.go:117] "RemoveContainer" containerID="1152859c2f36d2c326e152d170c7cac23dc3866a2123314ab30d753ad6cbf625"
Sep 11 22:07:00.358993 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:00.358969    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"]
Sep 11 22:07:00.365689 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:00.365657    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7d5ec8-dc5cdcc46-4rvkr"]
Sep 11 22:07:00.380832 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:00.380799    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"]
Sep 11 22:07:00.384286 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:00.384256    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7d5ec8-694b66b5d9-48c7d"]
Sep 11 22:07:01.005372 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:01.005338    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:01.720138 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:01.720102    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2246aba-6a16-4848-9509-5fa42eb93d3b" path="/var/lib/kubelet/pods/c2246aba-6a16-4848-9509-5fa42eb93d3b/volumes"
Sep 11 22:07:01.720575 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:01.720555    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5aed87d-88ba-46ff-93e2-d2a658f32b62" path="/var/lib/kubelet/pods/d5aed87d-88ba-46ff-93e2-d2a658f32b62/volumes"
Sep 11 22:07:01.818102 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:01.818068    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:02.501118 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:02.501085    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:03.179375 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:03.179305    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:03.851674 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:03.851646    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"]
Sep 11 22:07:03.852099 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:03.851849    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq" podUID="4bd34b82-d7d0-4e92-839f-6e6a92a6afd3" containerName="maas-api" containerID="cri-o://0cc412ba167c04a5a4dbc337adf7555df39693c91213debdbddfe357c51ae442" gracePeriod=30
Sep 11 22:07:03.881914 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:03.881887    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:03.909326 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:03.909268    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"]
Sep 11 22:07:03.909549 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:03.909512    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg" podUID="a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a" containerName="payload-processing" containerID="cri-o://43bb90a6669128459b49902d0f30ac0997c90d775937fb0aab15e61e2fdc2a01" gracePeriod=30
Sep 11 22:07:03.923947 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:03.923918    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"]
Sep 11 22:07:03.924165 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:03.924138    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh" podUID="4826bcc3-393b-4af4-ad7a-7a78ad11d838" containerName="payload-pre-processing" containerID="cri-o://68b5de5fdb2cb22e681f9c8abc8580de2eacce1d3f54e68418843c756891ef85" gracePeriod=30
Sep 11 22:07:03.960714 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:03.960686    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:04.033053 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.033021    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:04.090241 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.089846    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:04.198869 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.198839    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:07:04.204988 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.204961    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"
Sep 11 22:07:04.221382 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.220545    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-openshift-service-ca\") pod \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") "
Sep 11 22:07:04.221382 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.220591    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-metrics-certs\") pod \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") "
Sep 11 22:07:04.221382 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.220622    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a-plugins-config-volume\") pod \"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a\" (UID: \"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a\") "
Sep 11 22:07:04.221382 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.220664    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hklg7\" (UniqueName: \"kubernetes.io/projected/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-kube-api-access-hklg7\") pod \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") "
Sep 11 22:07:04.221382 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.220694    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-maas-api-tls\") pod \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\" (UID: \"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3\") "
Sep 11 22:07:04.221382 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.220743    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl6f2\" (UniqueName: \"kubernetes.io/projected/a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a-kube-api-access-jl6f2\") pod \"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a\" (UID: \"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a\") "
Sep 11 22:07:04.222274 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.221801    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4bd34b82-d7d0-4e92-839f-6e6a92a6afd3" (UID: "4bd34b82-d7d0-4e92-839f-6e6a92a6afd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:07:04.222274 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.222219    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a" (UID: "a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:07:04.229637 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.229257    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4bd34b82-d7d0-4e92-839f-6e6a92a6afd3" (UID: "4bd34b82-d7d0-4e92-839f-6e6a92a6afd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:07:04.229637 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.229289    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a-kube-api-access-jl6f2" (OuterVolumeSpecName: "kube-api-access-jl6f2") pod "a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a" (UID: "a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a"). InnerVolumeSpecName "kube-api-access-jl6f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:07:04.229946 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.229915    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-kube-api-access-hklg7" (OuterVolumeSpecName: "kube-api-access-hklg7") pod "4bd34b82-d7d0-4e92-839f-6e6a92a6afd3" (UID: "4bd34b82-d7d0-4e92-839f-6e6a92a6afd3"). InnerVolumeSpecName "kube-api-access-hklg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:07:04.231134 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.231098    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4bd34b82-d7d0-4e92-839f-6e6a92a6afd3" (UID: "4bd34b82-d7d0-4e92-839f-6e6a92a6afd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:07:04.249477 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.249423    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:04.259546 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.259518    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"
Sep 11 22:07:04.322504 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.322466    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9f5f\" (UniqueName: \"kubernetes.io/projected/4826bcc3-393b-4af4-ad7a-7a78ad11d838-kube-api-access-d9f5f\") pod \"4826bcc3-393b-4af4-ad7a-7a78ad11d838\" (UID: \"4826bcc3-393b-4af4-ad7a-7a78ad11d838\") "
Sep 11 22:07:04.322688 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.322536    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4826bcc3-393b-4af4-ad7a-7a78ad11d838-plugins-config-volume\") pod \"4826bcc3-393b-4af4-ad7a-7a78ad11d838\" (UID: \"4826bcc3-393b-4af4-ad7a-7a78ad11d838\") "
Sep 11 22:07:04.322881 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.322861    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-metrics-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:07:04.323028 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.322874    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4826bcc3-393b-4af4-ad7a-7a78ad11d838-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4826bcc3-393b-4af4-ad7a-7a78ad11d838" (UID: "4826bcc3-393b-4af4-ad7a-7a78ad11d838"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:07:04.323028 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.322884    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:07:04.323028 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.322934    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hklg7\" (UniqueName: \"kubernetes.io/projected/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-kube-api-access-hklg7\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:07:04.323028 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.322954    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-maas-api-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:07:04.323028 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.322969    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl6f2\" (UniqueName: \"kubernetes.io/projected/a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a-kube-api-access-jl6f2\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:07:04.323028 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.322982    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3-openshift-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:07:04.324670 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.324643    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4826bcc3-393b-4af4-ad7a-7a78ad11d838-kube-api-access-d9f5f" (OuterVolumeSpecName: "kube-api-access-d9f5f") pod "4826bcc3-393b-4af4-ad7a-7a78ad11d838" (UID: "4826bcc3-393b-4af4-ad7a-7a78ad11d838"). InnerVolumeSpecName "kube-api-access-d9f5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:07:04.362971 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.362881    2574 generic.go:356] "Generic (PLEG): container finished" podID="4bd34b82-d7d0-4e92-839f-6e6a92a6afd3" containerID="0cc412ba167c04a5a4dbc337adf7555df39693c91213debdbddfe357c51ae442" exitCode=0
Sep 11 22:07:04.362971 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.362945    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"
Sep 11 22:07:04.363201 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.362974    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq" event={"ID":"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3","Type":"ContainerDied","Data":"0cc412ba167c04a5a4dbc337adf7555df39693c91213debdbddfe357c51ae442"}
Sep 11 22:07:04.363201 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.363047    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq" event={"ID":"4bd34b82-d7d0-4e92-839f-6e6a92a6afd3","Type":"ContainerDied","Data":"456315324c71569ec24a07e9cca008004362bfe898c5ea92239ddd2503841779"}
Sep 11 22:07:04.363201 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.363094    2574 scope.go:117] "RemoveContainer" containerID="0cc412ba167c04a5a4dbc337adf7555df39693c91213debdbddfe357c51ae442"
Sep 11 22:07:04.365010 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.364985    2574 generic.go:356] "Generic (PLEG): container finished" podID="a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a" containerID="43bb90a6669128459b49902d0f30ac0997c90d775937fb0aab15e61e2fdc2a01" exitCode=0
Sep 11 22:07:04.365118 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.365069    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg" event={"ID":"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a","Type":"ContainerDied","Data":"43bb90a6669128459b49902d0f30ac0997c90d775937fb0aab15e61e2fdc2a01"}
Sep 11 22:07:04.365180 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.365136    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg" event={"ID":"a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a","Type":"ContainerDied","Data":"aa67391516a47c759e26a449d1cdd6ece2138872459689de2f62f7036aafc4a7"}
Sep 11 22:07:04.365180 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.365077    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"
Sep 11 22:07:04.366792 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.366769    2574 generic.go:356] "Generic (PLEG): container finished" podID="4826bcc3-393b-4af4-ad7a-7a78ad11d838" containerID="68b5de5fdb2cb22e681f9c8abc8580de2eacce1d3f54e68418843c756891ef85" exitCode=0
Sep 11 22:07:04.366910 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.366825    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh" event={"ID":"4826bcc3-393b-4af4-ad7a-7a78ad11d838","Type":"ContainerDied","Data":"68b5de5fdb2cb22e681f9c8abc8580de2eacce1d3f54e68418843c756891ef85"}
Sep 11 22:07:04.366910 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.366864    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh" event={"ID":"4826bcc3-393b-4af4-ad7a-7a78ad11d838","Type":"ContainerDied","Data":"0faae05c786c8381257791181780aa0efe071989685456a3576a3862bdaf5a31"}
Sep 11 22:07:04.366910 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.366870    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"
Sep 11 22:07:04.374591 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.374575    2574 scope.go:117] "RemoveContainer" containerID="0cc412ba167c04a5a4dbc337adf7555df39693c91213debdbddfe357c51ae442"
Sep 11 22:07:04.374899 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:07:04.374878    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc412ba167c04a5a4dbc337adf7555df39693c91213debdbddfe357c51ae442\": container with ID starting with 0cc412ba167c04a5a4dbc337adf7555df39693c91213debdbddfe357c51ae442 not found: ID does not exist" containerID="0cc412ba167c04a5a4dbc337adf7555df39693c91213debdbddfe357c51ae442"
Sep 11 22:07:04.374998 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.374907    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc412ba167c04a5a4dbc337adf7555df39693c91213debdbddfe357c51ae442"} err="failed to get container status \"0cc412ba167c04a5a4dbc337adf7555df39693c91213debdbddfe357c51ae442\": rpc error: code = NotFound desc = could not find container \"0cc412ba167c04a5a4dbc337adf7555df39693c91213debdbddfe357c51ae442\": container with ID starting with 0cc412ba167c04a5a4dbc337adf7555df39693c91213debdbddfe357c51ae442 not found: ID does not exist"
Sep 11 22:07:04.374998 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.374931    2574 scope.go:117] "RemoveContainer" containerID="43bb90a6669128459b49902d0f30ac0997c90d775937fb0aab15e61e2fdc2a01"
Sep 11 22:07:04.383282 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.383266    2574 scope.go:117] "RemoveContainer" containerID="43bb90a6669128459b49902d0f30ac0997c90d775937fb0aab15e61e2fdc2a01"
Sep 11 22:07:04.383591 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:07:04.383571    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43bb90a6669128459b49902d0f30ac0997c90d775937fb0aab15e61e2fdc2a01\": container with ID starting with 43bb90a6669128459b49902d0f30ac0997c90d775937fb0aab15e61e2fdc2a01 not found: ID does not exist" containerID="43bb90a6669128459b49902d0f30ac0997c90d775937fb0aab15e61e2fdc2a01"
Sep 11 22:07:04.383656 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.383599    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43bb90a6669128459b49902d0f30ac0997c90d775937fb0aab15e61e2fdc2a01"} err="failed to get container status \"43bb90a6669128459b49902d0f30ac0997c90d775937fb0aab15e61e2fdc2a01\": rpc error: code = NotFound desc = could not find container \"43bb90a6669128459b49902d0f30ac0997c90d775937fb0aab15e61e2fdc2a01\": container with ID starting with 43bb90a6669128459b49902d0f30ac0997c90d775937fb0aab15e61e2fdc2a01 not found: ID does not exist"
Sep 11 22:07:04.383656 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.383616    2574 scope.go:117] "RemoveContainer" containerID="68b5de5fdb2cb22e681f9c8abc8580de2eacce1d3f54e68418843c756891ef85"
Sep 11 22:07:04.390264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.390241    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"]
Sep 11 22:07:04.392082 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.392066    2574 scope.go:117] "RemoveContainer" containerID="68b5de5fdb2cb22e681f9c8abc8580de2eacce1d3f54e68418843c756891ef85"
Sep 11 22:07:04.392248 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.392229    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c9db036-6bc674568f-n9qjq"]
Sep 11 22:07:04.392396 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:07:04.392368    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b5de5fdb2cb22e681f9c8abc8580de2eacce1d3f54e68418843c756891ef85\": container with ID starting with 68b5de5fdb2cb22e681f9c8abc8580de2eacce1d3f54e68418843c756891ef85 not found: ID does not exist" containerID="68b5de5fdb2cb22e681f9c8abc8580de2eacce1d3f54e68418843c756891ef85"
Sep 11 22:07:04.392456 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.392396    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b5de5fdb2cb22e681f9c8abc8580de2eacce1d3f54e68418843c756891ef85"} err="failed to get container status \"68b5de5fdb2cb22e681f9c8abc8580de2eacce1d3f54e68418843c756891ef85\": rpc error: code = NotFound desc = could not find container \"68b5de5fdb2cb22e681f9c8abc8580de2eacce1d3f54e68418843c756891ef85\": container with ID starting with 68b5de5fdb2cb22e681f9c8abc8580de2eacce1d3f54e68418843c756891ef85 not found: ID does not exist"
Sep 11 22:07:04.402557 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.402533    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"]
Sep 11 22:07:04.411492 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.408938    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2c9db036-849f7b8f57-p8vfg"]
Sep 11 22:07:04.422503 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.422484    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"]
Sep 11 22:07:04.423443 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.423421    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9f5f\" (UniqueName: \"kubernetes.io/projected/4826bcc3-393b-4af4-ad7a-7a78ad11d838-kube-api-access-d9f5f\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:07:04.423443 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.423443    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4826bcc3-393b-4af4-ad7a-7a78ad11d838-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:07:04.427597 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.427574    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2c9db036-866d5dcf74-hfsgh"]
Sep 11 22:07:04.870707 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.870682    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:04.975709 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:04.975680    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:05.080843 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:05.080801    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:05.171176 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:05.171095    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:05.720151 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:05.720116    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4826bcc3-393b-4af4-ad7a-7a78ad11d838" path="/var/lib/kubelet/pods/4826bcc3-393b-4af4-ad7a-7a78ad11d838/volumes"
Sep 11 22:07:05.720567 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:05.720549    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bd34b82-d7d0-4e92-839f-6e6a92a6afd3" path="/var/lib/kubelet/pods/4bd34b82-d7d0-4e92-839f-6e6a92a6afd3/volumes"
Sep 11 22:07:05.720878 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:05.720865    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a" path="/var/lib/kubelet/pods/a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a/volumes"
Sep 11 22:07:05.980897 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:05.980816    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:06.485672 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:06.485636    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:06.795860 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:06.795775    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:07.475607 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:07.475571    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:09.013906 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:09.013869    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:12.231248 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:12.231212    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:12.388274 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:12.388238    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:12.476997 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:12.476958    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:12.577295 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:12.577202    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:12.933097 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:12.933061    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:13.703585 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:13.703558    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:07:13.704597 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:13.704575    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:07:13.708155 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:13.708135    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:07:13.708983 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:13.708962    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:07:22.380644 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.380602    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"]
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381280    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6e8c0e1-bd44-4116-8d00-2305b02a2740" containerName="revoke-keys"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381302    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6e8c0e1-bd44-4116-8d00-2305b02a2740" containerName="revoke-keys"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381378    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5aed87d-88ba-46ff-93e2-d2a658f32b62" containerName="payload-processing"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381387    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5aed87d-88ba-46ff-93e2-d2a658f32b62" containerName="payload-processing"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381406    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2246aba-6a16-4848-9509-5fa42eb93d3b" containerName="payload-pre-processing"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381414    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2246aba-6a16-4848-9509-5fa42eb93d3b" containerName="payload-pre-processing"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381425    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a" containerName="payload-processing"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381434    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a" containerName="payload-processing"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381451    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bd34b82-d7d0-4e92-839f-6e6a92a6afd3" containerName="maas-api"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381459    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bd34b82-d7d0-4e92-839f-6e6a92a6afd3" containerName="maas-api"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381498    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4826bcc3-393b-4af4-ad7a-7a78ad11d838" containerName="payload-pre-processing"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381507    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4826bcc3-393b-4af4-ad7a-7a78ad11d838" containerName="payload-pre-processing"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381637    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2246aba-6a16-4848-9509-5fa42eb93d3b" containerName="payload-pre-processing"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381650    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4826bcc3-393b-4af4-ad7a-7a78ad11d838" containerName="payload-pre-processing"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381661    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a67dd6ca-b6cd-47fc-93dc-3fdf41c7373a" containerName="payload-processing"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381671    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bd34b82-d7d0-4e92-839f-6e6a92a6afd3" containerName="maas-api"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381680    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6e8c0e1-bd44-4116-8d00-2305b02a2740" containerName="revoke-keys"
Sep 11 22:07:22.383358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.381688    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5aed87d-88ba-46ff-93e2-d2a658f32b62" containerName="payload-processing"
Sep 11 22:07:22.385061 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.385041    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.389424 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.389397    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7e2add\"/\"premium-w4-7e2add-kserve-self-signed-certs\""
Sep 11 22:07:22.389578 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.389446    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7e2add\"/\"default-dockercfg-2r4jm\""
Sep 11 22:07:22.389578 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.389481    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7e2add\"/\"kube-root-ca.crt\""
Sep 11 22:07:22.389578 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.389492    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7e2add\"/\"openshift-service-ca.crt\""
Sep 11 22:07:22.392915 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.392893    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"]
Sep 11 22:07:22.399181 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.399141    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:22.485124 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.485088    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdzjm\" (UniqueName: \"kubernetes.io/projected/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-kube-api-access-kdzjm\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.485295 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.485136    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-model-cache\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.485295 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.485190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-tls-certs\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.485295 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.485225    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-tmp-dir\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.485295 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.485242    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-dshm\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.485295 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.485261    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-home\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.586290 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.586255    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-tmp-dir\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.586290 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.586291    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-dshm\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.586540 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.586323    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-home\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.586540 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.586371    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdzjm\" (UniqueName: \"kubernetes.io/projected/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-kube-api-access-kdzjm\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.586540 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.586410    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-model-cache\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.586540 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.586462    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-tls-certs\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.586825 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.586794    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-home\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.586952 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.586839    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-tmp-dir\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.586952 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.586846    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-model-cache\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.588570 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.588543    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-dshm\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.588810 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.588794    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-tls-certs\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.594898 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.594875    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdzjm\" (UniqueName: \"kubernetes.io/projected/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-kube-api-access-kdzjm\") pod \"premium-w4-7e2add-kserve-7f5f4c877f-6mqbq\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") " pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.701038 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.700944    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:22.972429 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:22.972403    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"]
Sep 11 22:07:22.974190 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:07:22.974166    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod847ba169_10a6_453f_9cd2_ca9ea2dfe24d.slice/crio-e230dea70bf7e779123fc87a8575c9a3df049ffd94f73feba98b978bde31f88b WatchSource:0}: Error finding container e230dea70bf7e779123fc87a8575c9a3df049ffd94f73feba98b978bde31f88b: Status 404 returned error can't find the container with id e230dea70bf7e779123fc87a8575c9a3df049ffd94f73feba98b978bde31f88b
Sep 11 22:07:23.481760 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:23.481729    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq" event={"ID":"847ba169-10a6-453f-9cd2-ca9ea2dfe24d","Type":"ContainerStarted","Data":"4e81acc3b8ab8ac4335a8ff53c3020641e970eb9c2e187d1052dc0c34cd7428f"}
Sep 11 22:07:23.482145 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:23.481778    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq" event={"ID":"847ba169-10a6-453f-9cd2-ca9ea2dfe24d","Type":"ContainerStarted","Data":"e230dea70bf7e779123fc87a8575c9a3df049ffd94f73feba98b978bde31f88b"}
Sep 11 22:07:23.499538 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:23.499490    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq" podStartSLOduration=1.49947634 podStartE2EDuration="1.49947634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:07:23.497914715 +0000 UTC m=+910.233026062" watchObservedRunningTime="2026-09-11 22:07:23.49947634 +0000 UTC m=+910.234587686"
Sep 11 22:07:29.876913 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.876883    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"]
Sep 11 22:07:29.880415 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.880396    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:29.884610 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.884588    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-95d6c2\"/\"kube-root-ca.crt\""
Sep 11 22:07:29.884610 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.884603    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-95d6c2\"/\"premium-w0-95d6c2-kserve-self-signed-certs\""
Sep 11 22:07:29.884796 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.884629    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-95d6c2\"/\"openshift-service-ca.crt\""
Sep 11 22:07:29.884796 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.884588    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-95d6c2\"/\"default-dockercfg-xs9lg\""
Sep 11 22:07:29.889329 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.889288    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"]
Sep 11 22:07:29.895188 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.895169    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:29.956240 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.956211    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-dshm\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:29.956396 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.956261    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlmkh\" (UniqueName: \"kubernetes.io/projected/d291c1a9-5ba8-4c01-8509-9e2c1c911134-kube-api-access-xlmkh\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:29.956396 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.956368    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-home\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:29.956396 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.956390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d291c1a9-5ba8-4c01-8509-9e2c1c911134-tls-certs\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:29.956522 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.956408    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-model-cache\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:29.956522 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:29.956432    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-tmp-dir\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.057228 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.057195    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-dshm\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.057388 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.057253    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlmkh\" (UniqueName: \"kubernetes.io/projected/d291c1a9-5ba8-4c01-8509-9e2c1c911134-kube-api-access-xlmkh\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.057453 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.057409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-home\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.057453 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.057445    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d291c1a9-5ba8-4c01-8509-9e2c1c911134-tls-certs\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.057540 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.057477    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-model-cache\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.057540 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.057520    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-tmp-dir\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.057785 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.057758    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-home\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.057871 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.057810    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-model-cache\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.057871 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.057823    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-tmp-dir\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.059709 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.059685    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-dshm\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.059934 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.059911    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d291c1a9-5ba8-4c01-8509-9e2c1c911134-tls-certs\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.070850 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.070817    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlmkh\" (UniqueName: \"kubernetes.io/projected/d291c1a9-5ba8-4c01-8509-9e2c1c911134-kube-api-access-xlmkh\") pod \"premium-w0-95d6c2-kserve-6995c8b755-dqk95\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") " pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.190506 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.190430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:30.460019 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.459994    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"]
Sep 11 22:07:30.462119 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:07:30.462093    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd291c1a9_5ba8_4c01_8509_9e2c1c911134.slice/crio-f06a112e783c47ad7057f57b10c2e4671bdde1d7f6f9b2dc6f9c3fc6165c1d82 WatchSource:0}: Error finding container f06a112e783c47ad7057f57b10c2e4671bdde1d7f6f9b2dc6f9c3fc6165c1d82: Status 404 returned error can't find the container with id f06a112e783c47ad7057f57b10c2e4671bdde1d7f6f9b2dc6f9c3fc6165c1d82
Sep 11 22:07:30.521002 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:30.520976    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95" event={"ID":"d291c1a9-5ba8-4c01-8509-9e2c1c911134","Type":"ContainerStarted","Data":"f06a112e783c47ad7057f57b10c2e4671bdde1d7f6f9b2dc6f9c3fc6165c1d82"}
Sep 11 22:07:31.528181 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:31.528139    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95" event={"ID":"d291c1a9-5ba8-4c01-8509-9e2c1c911134","Type":"ContainerStarted","Data":"27288e3246092a101f43c436665c9d890e800220d62f6f674481d08bfca22154"}
Sep 11 22:07:31.545959 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:31.545910    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95" podStartSLOduration=2.545893 podStartE2EDuration="2.545893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:07:31.544062051 +0000 UTC m=+918.279173410" watchObservedRunningTime="2026-09-11 22:07:31.545893 +0000 UTC m=+918.281004345"
Sep 11 22:07:32.701738 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:32.701698    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:32.702203 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:32.701752    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:32.714737 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:32.714709    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:33.549983 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:33.549951    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:07:38.866487 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:38.866449    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:40.190699 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:40.190663    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:40.191099 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:40.190713    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:40.203219 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:40.203192    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:40.587736 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:40.587702    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:07:41.464939 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:41.464903    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:41.698476 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:41.698443    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:42.143892 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:42.143862    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:42.445341 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:42.445234    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:42.595446 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:42.595409    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:44.565498 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:44.565464    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:44.631662 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:44.631627    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:44.706038 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:44.706002    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:44.772874 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:44.772841    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:45.907878 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:45.907839    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:46.101133 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:46.101103    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:46.199691 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:46.199611    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:46.732415 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:46.732383    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:47.620848 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:47.620815    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:48.301707 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.301669    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:48.907925 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.907887    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"]
Sep 11 22:07:48.912468 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.912441    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:48.915229 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.915208    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-0aea78-serving-cert\""
Sep 11 22:07:48.915623 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.915603    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:07:48.916682 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.916663    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 22:07:48.916758 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.916715    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 22:07:48.917007 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.916986    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mc9f9\""
Sep 11 22:07:48.927761 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.927715    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"]
Sep 11 22:07:48.937486 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.937461    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-maas-api-tls\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:48.937579 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.937502    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a763b94b-084a-4543-806e-6ac72a75e2aa-openshift-service-ca\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:48.937579 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.937573    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfmnb\" (UniqueName: \"kubernetes.io/projected/a763b94b-084a-4543-806e-6ac72a75e2aa-kube-api-access-cfmnb\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:48.937666 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.937651    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-metrics-certs\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:48.945678 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:48.945644    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:49.038821 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.038787    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfmnb\" (UniqueName: \"kubernetes.io/projected/a763b94b-084a-4543-806e-6ac72a75e2aa-kube-api-access-cfmnb\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:49.038978 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.038835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-metrics-certs\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:49.038978 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.038943    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-maas-api-tls\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:49.038978 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.038965    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a763b94b-084a-4543-806e-6ac72a75e2aa-openshift-service-ca\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:49.039099 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:07:49.039066    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-serving-cert: secret "maas-api-e2e-explicit-0aea78-serving-cert" not found
Sep 11 22:07:49.039151 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:07:49.039139    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-maas-api-tls podName:a763b94b-084a-4543-806e-6ac72a75e2aa nodeName:}" failed. No retries permitted until 2026-09-11 22:07:49.539123924 +0000 UTC m=+936.274235252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-maas-api-tls") pod "maas-api-e2e-explicit-0aea78-84466784cf-mnzqb" (UID: "a763b94b-084a-4543-806e-6ac72a75e2aa") : secret "maas-api-e2e-explicit-0aea78-serving-cert" not found
Sep 11 22:07:49.039646 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.039626    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a763b94b-084a-4543-806e-6ac72a75e2aa-openshift-service-ca\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:49.041370 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.041353    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-metrics-certs\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:49.048300 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.048259    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfmnb\" (UniqueName: \"kubernetes.io/projected/a763b94b-084a-4543-806e-6ac72a75e2aa-kube-api-access-cfmnb\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:49.534497 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.534462    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"]
Sep 11 22:07:49.538045 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.538030    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"
Sep 11 22:07:49.540894 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.540861    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-0aea78\""
Sep 11 22:07:49.541030 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.540941    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-0aea78-dockercfg-d4zqt\""
Sep 11 22:07:49.543887 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.543866    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-maas-api-tls\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:49.544023 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:07:49.544002    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-serving-cert: secret "maas-api-e2e-explicit-0aea78-serving-cert" not found
Sep 11 22:07:49.544086 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:07:49.544076    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-maas-api-tls podName:a763b94b-084a-4543-806e-6ac72a75e2aa nodeName:}" failed. No retries permitted until 2026-09-11 22:07:50.544057477 +0000 UTC m=+937.279168803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-maas-api-tls") pod "maas-api-e2e-explicit-0aea78-84466784cf-mnzqb" (UID: "a763b94b-084a-4543-806e-6ac72a75e2aa") : secret "maas-api-e2e-explicit-0aea78-serving-cert" not found
Sep 11 22:07:49.548259 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.548236    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"]
Sep 11 22:07:49.644994 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.644964    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434c1fe0-78fb-411c-a3dc-cff8157ccd08-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4\" (UID: \"434c1fe0-78fb-411c-a3dc-cff8157ccd08\") " pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"
Sep 11 22:07:49.645155 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.645048    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc87t\" (UniqueName: \"kubernetes.io/projected/434c1fe0-78fb-411c-a3dc-cff8157ccd08-kube-api-access-sc87t\") pod \"payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4\" (UID: \"434c1fe0-78fb-411c-a3dc-cff8157ccd08\") " pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"
Sep 11 22:07:49.746629 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.746594    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434c1fe0-78fb-411c-a3dc-cff8157ccd08-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4\" (UID: \"434c1fe0-78fb-411c-a3dc-cff8157ccd08\") " pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"
Sep 11 22:07:49.746938 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.746918    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc87t\" (UniqueName: \"kubernetes.io/projected/434c1fe0-78fb-411c-a3dc-cff8157ccd08-kube-api-access-sc87t\") pod \"payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4\" (UID: \"434c1fe0-78fb-411c-a3dc-cff8157ccd08\") " pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"
Sep 11 22:07:49.748418 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.748391    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434c1fe0-78fb-411c-a3dc-cff8157ccd08-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4\" (UID: \"434c1fe0-78fb-411c-a3dc-cff8157ccd08\") " pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"
Sep 11 22:07:49.756101 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.756074    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc87t\" (UniqueName: \"kubernetes.io/projected/434c1fe0-78fb-411c-a3dc-cff8157ccd08-kube-api-access-sc87t\") pod \"payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4\" (UID: \"434c1fe0-78fb-411c-a3dc-cff8157ccd08\") " pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"
Sep 11 22:07:49.849932 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:49.849901    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"
Sep 11 22:07:50.125227 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:50.125197    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"]
Sep 11 22:07:50.128041 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:07:50.128008    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434c1fe0_78fb_411c_a3dc_cff8157ccd08.slice/crio-c68a205caf31fe5375a3a925d6c03c3d5d37833637a543e2987472573b1daf39 WatchSource:0}: Error finding container c68a205caf31fe5375a3a925d6c03c3d5d37833637a543e2987472573b1daf39: Status 404 returned error can't find the container with id c68a205caf31fe5375a3a925d6c03c3d5d37833637a543e2987472573b1daf39
Sep 11 22:07:50.554305 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:50.554220    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-maas-api-tls\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:50.556706 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:50.556678    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-maas-api-tls\") pod \"maas-api-e2e-explicit-0aea78-84466784cf-mnzqb\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:50.632357 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:50.632324    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4" event={"ID":"434c1fe0-78fb-411c-a3dc-cff8157ccd08","Type":"ContainerStarted","Data":"95a705b57112f98e2b7a64f5c45a5c2efc355c55e021d9a4054a22aba56acaf9"}
Sep 11 22:07:50.632495 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:50.632374    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4" event={"ID":"434c1fe0-78fb-411c-a3dc-cff8157ccd08","Type":"ContainerStarted","Data":"c68a205caf31fe5375a3a925d6c03c3d5d37833637a543e2987472573b1daf39"}
Sep 11 22:07:50.632495 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:50.632425    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"
Sep 11 22:07:50.649481 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:50.649437    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4" podStartSLOduration=1.649422875 podStartE2EDuration="1.649422875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:07:50.647779441 +0000 UTC m=+937.382890787" watchObservedRunningTime="2026-09-11 22:07:50.649422875 +0000 UTC m=+937.384534220"
Sep 11 22:07:50.725669 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:50.725641    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:50.745880 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:50.745844    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:51.003957 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:51.003923    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"]
Sep 11 22:07:51.006019 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:07:51.005991    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda763b94b_084a_4543_806e_6ac72a75e2aa.slice/crio-eb47064bf01b255d83469bb59f693d54535cc0d52d5127682780679ec97c8142 WatchSource:0}: Error finding container eb47064bf01b255d83469bb59f693d54535cc0d52d5127682780679ec97c8142: Status 404 returned error can't find the container with id eb47064bf01b255d83469bb59f693d54535cc0d52d5127682780679ec97c8142
Sep 11 22:07:51.007581 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:51.007561    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:07:51.640037 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:51.640005    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb" event={"ID":"a763b94b-084a-4543-806e-6ac72a75e2aa","Type":"ContainerStarted","Data":"64f5e04a1b0fd8086dc5ad846a4a52d5200c55f2858b940cbd7f31445f892970"}
Sep 11 22:07:51.640522 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:51.640061    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb" event={"ID":"a763b94b-084a-4543-806e-6ac72a75e2aa","Type":"ContainerStarted","Data":"eb47064bf01b255d83469bb59f693d54535cc0d52d5127682780679ec97c8142"}
Sep 11 22:07:51.640522 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:51.640138    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:07:51.658977 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:51.658931    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb" podStartSLOduration=3.243872716 podStartE2EDuration="3.658918637s" totalImagesPullingTime="415.045921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:07:51.657293112 +0000 UTC m=+938.392404452" watchObservedRunningTime="2026-09-11 22:07:51.658918637 +0000 UTC m=+938.394029984"
Sep 11 22:07:54.371636 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:54.371600    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:57.651500 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:07:57.651469    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:08:00.898961 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:00.898909    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:01.642591 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:01.642563    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"
Sep 11 22:08:06.648209 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:06.648171    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:06.758256 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:06.758221    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:06.847939 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:06.847900    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:11.036009 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:11.035969    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:11.136107 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:11.136065    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:11.805110 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:11.805073    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:12.544032 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:12.543997    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:13.199686 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:13.199653    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:13.893882 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:13.893850    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:14.538029 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:14.537980    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:15.094040 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:15.093990    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:15.194518 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:15.194471    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:17.356447 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:17.356411    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:18.122248 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:18.122215    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:20.158244 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:20.158208    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:23.938719 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:23.938685    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:24.004269 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:24.004221    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:24.090233 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:24.090208    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:24.189962 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:24.189887    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:24.872041 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:24.872004    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:24.971516 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:24.971469    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:25.071286 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:25.071249    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:26.613052 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:26.613017    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:32.485075 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:32.485039    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:32.585188 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:32.585132    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:33.172282 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:33.172243    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:38.837789 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:38.837748    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:38.946759 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:38.946705    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:39.098847 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:39.098768    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:39.385476 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:39.385366    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:39.495121 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:39.495081    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:44.602053 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:44.602025    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:44.820225 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:44.820188    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:44.912623 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:44.912533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:45.523290 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:45.523252    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:46.368490 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:46.368439    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:47.041670 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:47.041616    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:47.796888 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:47.796855    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:48.487536 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:48.487496    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:48.587780 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:48.587742    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:48.787973 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:48.787888    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:48.949443 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:48.949408    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:49.048692 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.048599    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:49.070147 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.070119    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"]
Sep 11 22:08:49.073821 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.073804    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"
Sep 11 22:08:49.076689 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.076670    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-27f26465\""
Sep 11 22:08:49.076793 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.076775    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-27f26465-dockercfg-ncslb\""
Sep 11 22:08:49.084352 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.084325    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"]
Sep 11 22:08:49.093209 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.093183    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"]
Sep 11 22:08:49.096708 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.096687    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"
Sep 11 22:08:49.111824 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.111802    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"]
Sep 11 22:08:49.228806 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.228772    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwcsh\" (UniqueName: \"kubernetes.io/projected/326b55e2-0d6c-418a-aa3e-d2c74e968ee4-kube-api-access-jwcsh\") pod \"payload-processing-e2e-mt-27f26465-746677745-2v7cg\" (UID: \"326b55e2-0d6c-418a-aa3e-d2c74e968ee4\") " pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"
Sep 11 22:08:49.228982 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.228829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfh9g\" (UniqueName: \"kubernetes.io/projected/9248058a-db49-48d5-bc59-e73532eb22e3-kube-api-access-rfh9g\") pod \"payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw\" (UID: \"9248058a-db49-48d5-bc59-e73532eb22e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"
Sep 11 22:08:49.228982 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.228896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9248058a-db49-48d5-bc59-e73532eb22e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw\" (UID: \"9248058a-db49-48d5-bc59-e73532eb22e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"
Sep 11 22:08:49.229059 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.229015    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326b55e2-0d6c-418a-aa3e-d2c74e968ee4-plugins-config-volume\") pod \"payload-processing-e2e-mt-27f26465-746677745-2v7cg\" (UID: \"326b55e2-0d6c-418a-aa3e-d2c74e968ee4\") " pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"
Sep 11 22:08:49.330261 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.330169    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326b55e2-0d6c-418a-aa3e-d2c74e968ee4-plugins-config-volume\") pod \"payload-processing-e2e-mt-27f26465-746677745-2v7cg\" (UID: \"326b55e2-0d6c-418a-aa3e-d2c74e968ee4\") " pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"
Sep 11 22:08:49.330261 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.330218    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwcsh\" (UniqueName: \"kubernetes.io/projected/326b55e2-0d6c-418a-aa3e-d2c74e968ee4-kube-api-access-jwcsh\") pod \"payload-processing-e2e-mt-27f26465-746677745-2v7cg\" (UID: \"326b55e2-0d6c-418a-aa3e-d2c74e968ee4\") " pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"
Sep 11 22:08:49.330261 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.330251    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfh9g\" (UniqueName: \"kubernetes.io/projected/9248058a-db49-48d5-bc59-e73532eb22e3-kube-api-access-rfh9g\") pod \"payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw\" (UID: \"9248058a-db49-48d5-bc59-e73532eb22e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"
Sep 11 22:08:49.330503 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.330271    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9248058a-db49-48d5-bc59-e73532eb22e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw\" (UID: \"9248058a-db49-48d5-bc59-e73532eb22e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"
Sep 11 22:08:49.330985 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.330965    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326b55e2-0d6c-418a-aa3e-d2c74e968ee4-plugins-config-volume\") pod \"payload-processing-e2e-mt-27f26465-746677745-2v7cg\" (UID: \"326b55e2-0d6c-418a-aa3e-d2c74e968ee4\") " pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"
Sep 11 22:08:49.331023 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.330998    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9248058a-db49-48d5-bc59-e73532eb22e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw\" (UID: \"9248058a-db49-48d5-bc59-e73532eb22e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"
Sep 11 22:08:49.338751 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.338721    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfh9g\" (UniqueName: \"kubernetes.io/projected/9248058a-db49-48d5-bc59-e73532eb22e3-kube-api-access-rfh9g\") pod \"payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw\" (UID: \"9248058a-db49-48d5-bc59-e73532eb22e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"
Sep 11 22:08:49.338882 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.338808    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwcsh\" (UniqueName: \"kubernetes.io/projected/326b55e2-0d6c-418a-aa3e-d2c74e968ee4-kube-api-access-jwcsh\") pod \"payload-processing-e2e-mt-27f26465-746677745-2v7cg\" (UID: \"326b55e2-0d6c-418a-aa3e-d2c74e968ee4\") " pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"
Sep 11 22:08:49.385862 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.385826    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"
Sep 11 22:08:49.407865 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.407837    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"
Sep 11 22:08:49.664398 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.664367    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"]
Sep 11 22:08:49.667112 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:08:49.667057    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9248058a_db49_48d5_bc59_e73532eb22e3.slice/crio-fa5f5ad3aa993ccfe4b0df6b148c7c59b4283d9f3887b3542417cddad61b4417 WatchSource:0}: Error finding container fa5f5ad3aa993ccfe4b0df6b148c7c59b4283d9f3887b3542417cddad61b4417: Status 404 returned error can't find the container with id fa5f5ad3aa993ccfe4b0df6b148c7c59b4283d9f3887b3542417cddad61b4417
Sep 11 22:08:49.692917 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.692895    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"]
Sep 11 22:08:49.700555 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:08:49.700524    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326b55e2_0d6c_418a_aa3e_d2c74e968ee4.slice/crio-6ca3437153a84d958b48ba58629c02f62fb885bd5bf5adc9125a921cf0912a51 WatchSource:0}: Error finding container 6ca3437153a84d958b48ba58629c02f62fb885bd5bf5adc9125a921cf0912a51: Status 404 returned error can't find the container with id 6ca3437153a84d958b48ba58629c02f62fb885bd5bf5adc9125a921cf0912a51
Sep 11 22:08:49.966599 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.966509    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg" event={"ID":"326b55e2-0d6c-418a-aa3e-d2c74e968ee4","Type":"ContainerStarted","Data":"62fc1b8ce81bb71722bcd95ae80926020381a6ca0879db6b16311f686fd6afea"}
Sep 11 22:08:49.966599 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.966595    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg" event={"ID":"326b55e2-0d6c-418a-aa3e-d2c74e968ee4","Type":"ContainerStarted","Data":"6ca3437153a84d958b48ba58629c02f62fb885bd5bf5adc9125a921cf0912a51"}
Sep 11 22:08:49.967068 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.966627    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"
Sep 11 22:08:49.968231 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.968205    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw" event={"ID":"9248058a-db49-48d5-bc59-e73532eb22e3","Type":"ContainerStarted","Data":"f6e6e04e46b201e91d73eaa9f4f561cbe7ad8efd195b95488523a0d44e78c763"}
Sep 11 22:08:49.968374 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.968258    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw" event={"ID":"9248058a-db49-48d5-bc59-e73532eb22e3","Type":"ContainerStarted","Data":"fa5f5ad3aa993ccfe4b0df6b148c7c59b4283d9f3887b3542417cddad61b4417"}
Sep 11 22:08:49.968374 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.968286    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"
Sep 11 22:08:49.985073 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.985029    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg" podStartSLOduration=0.985014941 podStartE2EDuration="985.014941ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:08:49.982404025 +0000 UTC m=+996.717515373" watchObservedRunningTime="2026-09-11 22:08:49.985014941 +0000 UTC m=+996.720126286"
Sep 11 22:08:49.998817 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:49.998773    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw" podStartSLOduration=0.998760083 podStartE2EDuration="998.760083ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:08:49.997095194 +0000 UTC m=+996.732206556" watchObservedRunningTime="2026-09-11 22:08:49.998760083 +0000 UTC m=+996.733871428"
Sep 11 22:08:57.073696 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:08:57.073661    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:00.976165 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:00.976135    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"
Sep 11 22:09:00.976634 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:00.976293    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"
Sep 11 22:09:01.610636 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:01.610597    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:10.923656 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:10.923625    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:11.645324 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:11.645273    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:12.533028 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:12.532988    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:13.349488 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:13.349449    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:14.203564 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:14.203524    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:14.887220 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:14.887184    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:14.992806 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:14.992773    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:15.592659 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:15.592628    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:15.690099 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:15.690066    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:15.976927 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:15.976850    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:16.077429 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:16.077391    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:21.594976 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:21.594932    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:21.697768 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:21.697734    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:22.997240 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:22.997199    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:23.833712 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:23.833674    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:24.545531 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:24.545497    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:25.008414 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:25.008384    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:25.401687 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:25.401648    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:26.144893 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:26.144864    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:26.943436 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:26.943397    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:27.687261 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:27.687231    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:27.820272 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:27.820238    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:28.729219 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:28.729187    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:29.405292 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:29.405257    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:30.071379 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:30.071340    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:30.928629 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:30.928593    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:31.619418 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:31.619385    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:32.314691 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:32.314660    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:33.002094 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.002057    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:33.033153 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.033112    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"]
Sep 11 22:09:33.033391 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.033362    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg" podUID="326b55e2-0d6c-418a-aa3e-d2c74e968ee4" containerName="payload-processing" containerID="cri-o://62fc1b8ce81bb71722bcd95ae80926020381a6ca0879db6b16311f686fd6afea" gracePeriod=30
Sep 11 22:09:33.042244 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.042216    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"]
Sep 11 22:09:33.042554 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.042477    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw" podUID="9248058a-db49-48d5-bc59-e73532eb22e3" containerName="payload-pre-processing" containerID="cri-o://f6e6e04e46b201e91d73eaa9f4f561cbe7ad8efd195b95488523a0d44e78c763" gracePeriod=30
Sep 11 22:09:33.075157 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.075081    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:33.134245 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.134210    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:33.198327 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.198281    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:33.221463 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.217434    2574 generic.go:356] "Generic (PLEG): container finished" podID="326b55e2-0d6c-418a-aa3e-d2c74e968ee4" containerID="62fc1b8ce81bb71722bcd95ae80926020381a6ca0879db6b16311f686fd6afea" exitCode=0
Sep 11 22:09:33.221463 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.217477    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg" event={"ID":"326b55e2-0d6c-418a-aa3e-d2c74e968ee4","Type":"ContainerDied","Data":"62fc1b8ce81bb71722bcd95ae80926020381a6ca0879db6b16311f686fd6afea"}
Sep 11 22:09:33.221463 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.219830    2574 generic.go:356] "Generic (PLEG): container finished" podID="9248058a-db49-48d5-bc59-e73532eb22e3" containerID="f6e6e04e46b201e91d73eaa9f4f561cbe7ad8efd195b95488523a0d44e78c763" exitCode=0
Sep 11 22:09:33.221463 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.219915    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw" event={"ID":"9248058a-db49-48d5-bc59-e73532eb22e3","Type":"ContainerDied","Data":"f6e6e04e46b201e91d73eaa9f4f561cbe7ad8efd195b95488523a0d44e78c763"}
Sep 11 22:09:33.258725 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.258639    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:33.293383 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.293358    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"
Sep 11 22:09:33.339599 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.339569    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"
Sep 11 22:09:33.431971 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.431935    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfh9g\" (UniqueName: \"kubernetes.io/projected/9248058a-db49-48d5-bc59-e73532eb22e3-kube-api-access-rfh9g\") pod \"9248058a-db49-48d5-bc59-e73532eb22e3\" (UID: \"9248058a-db49-48d5-bc59-e73532eb22e3\") "
Sep 11 22:09:33.432150 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.432005    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326b55e2-0d6c-418a-aa3e-d2c74e968ee4-plugins-config-volume\") pod \"326b55e2-0d6c-418a-aa3e-d2c74e968ee4\" (UID: \"326b55e2-0d6c-418a-aa3e-d2c74e968ee4\") "
Sep 11 22:09:33.432150 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.432072    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwcsh\" (UniqueName: \"kubernetes.io/projected/326b55e2-0d6c-418a-aa3e-d2c74e968ee4-kube-api-access-jwcsh\") pod \"326b55e2-0d6c-418a-aa3e-d2c74e968ee4\" (UID: \"326b55e2-0d6c-418a-aa3e-d2c74e968ee4\") "
Sep 11 22:09:33.432150 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.432096    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9248058a-db49-48d5-bc59-e73532eb22e3-plugins-config-volume\") pod \"9248058a-db49-48d5-bc59-e73532eb22e3\" (UID: \"9248058a-db49-48d5-bc59-e73532eb22e3\") "
Sep 11 22:09:33.432541 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.432516    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326b55e2-0d6c-418a-aa3e-d2c74e968ee4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "326b55e2-0d6c-418a-aa3e-d2c74e968ee4" (UID: "326b55e2-0d6c-418a-aa3e-d2c74e968ee4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:09:33.432649 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.432524    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9248058a-db49-48d5-bc59-e73532eb22e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9248058a-db49-48d5-bc59-e73532eb22e3" (UID: "9248058a-db49-48d5-bc59-e73532eb22e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:09:33.434189 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.434171    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326b55e2-0d6c-418a-aa3e-d2c74e968ee4-kube-api-access-jwcsh" (OuterVolumeSpecName: "kube-api-access-jwcsh") pod "326b55e2-0d6c-418a-aa3e-d2c74e968ee4" (UID: "326b55e2-0d6c-418a-aa3e-d2c74e968ee4"). InnerVolumeSpecName "kube-api-access-jwcsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:09:33.434266 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.434210    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9248058a-db49-48d5-bc59-e73532eb22e3-kube-api-access-rfh9g" (OuterVolumeSpecName: "kube-api-access-rfh9g") pod "9248058a-db49-48d5-bc59-e73532eb22e3" (UID: "9248058a-db49-48d5-bc59-e73532eb22e3"). InnerVolumeSpecName "kube-api-access-rfh9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:09:33.533600 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.533524    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfh9g\" (UniqueName: \"kubernetes.io/projected/9248058a-db49-48d5-bc59-e73532eb22e3-kube-api-access-rfh9g\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:09:33.533600 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.533548    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326b55e2-0d6c-418a-aa3e-d2c74e968ee4-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:09:33.533600 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.533557    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwcsh\" (UniqueName: \"kubernetes.io/projected/326b55e2-0d6c-418a-aa3e-d2c74e968ee4-kube-api-access-jwcsh\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:09:33.533600 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.533568    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9248058a-db49-48d5-bc59-e73532eb22e3-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:09:33.549751 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.549707    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:33.650018 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:33.649984    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:34.227813 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:34.227782    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"
Sep 11 22:09:34.227813 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:34.227806    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg" event={"ID":"326b55e2-0d6c-418a-aa3e-d2c74e968ee4","Type":"ContainerDied","Data":"6ca3437153a84d958b48ba58629c02f62fb885bd5bf5adc9125a921cf0912a51"}
Sep 11 22:09:34.228363 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:34.227871    2574 scope.go:117] "RemoveContainer" containerID="62fc1b8ce81bb71722bcd95ae80926020381a6ca0879db6b16311f686fd6afea"
Sep 11 22:09:34.229433 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:34.229396    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"
Sep 11 22:09:34.229559 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:34.229429    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw" event={"ID":"9248058a-db49-48d5-bc59-e73532eb22e3","Type":"ContainerDied","Data":"fa5f5ad3aa993ccfe4b0df6b148c7c59b4283d9f3887b3542417cddad61b4417"}
Sep 11 22:09:34.237571 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:34.237551    2574 scope.go:117] "RemoveContainer" containerID="f6e6e04e46b201e91d73eaa9f4f561cbe7ad8efd195b95488523a0d44e78c763"
Sep 11 22:09:34.248723 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:34.248701    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"]
Sep 11 22:09:34.251077 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:34.251054    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27f26465-56b8694dbb-r74tw"]
Sep 11 22:09:34.265326 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:34.264617    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"]
Sep 11 22:09:34.269583 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:34.269559    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27f26465-746677745-2v7cg"]
Sep 11 22:09:34.354434 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:34.354398    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:35.105479 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:35.105440    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:35.721849 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:35.721819    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="326b55e2-0d6c-418a-aa3e-d2c74e968ee4" path="/var/lib/kubelet/pods/326b55e2-0d6c-418a-aa3e-d2c74e968ee4/volumes"
Sep 11 22:09:35.722215 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:35.722181    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9248058a-db49-48d5-bc59-e73532eb22e3" path="/var/lib/kubelet/pods/9248058a-db49-48d5-bc59-e73532eb22e3/volumes"
Sep 11 22:09:35.908385 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:35.908350    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:36.574505 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:36.574451    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:37.369834 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:37.369792    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:37.906111 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:37.906076    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"]
Sep 11 22:09:37.906330 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:37.906284    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb" podUID="a763b94b-084a-4543-806e-6ac72a75e2aa" containerName="maas-api" containerID="cri-o://64f5e04a1b0fd8086dc5ad846a4a52d5200c55f2858b940cbd7f31445f892970" gracePeriod=30
Sep 11 22:09:37.931983 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:37.931960    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:37.964345 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:37.964320    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"]
Sep 11 22:09:37.964544 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:37.964520    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4" podUID="434c1fe0-78fb-411c-a3dc-cff8157ccd08" containerName="payload-processing" containerID="cri-o://95a705b57112f98e2b7a64f5c45a5c2efc355c55e021d9a4054a22aba56acaf9" gracePeriod=30
Sep 11 22:09:38.002518 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.002420    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:38.104501 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.103093    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:38.158276 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.158206    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:38.254236 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.254210    2574 generic.go:356] "Generic (PLEG): container finished" podID="434c1fe0-78fb-411c-a3dc-cff8157ccd08" containerID="95a705b57112f98e2b7a64f5c45a5c2efc355c55e021d9a4054a22aba56acaf9" exitCode=0
Sep 11 22:09:38.254381 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.254304    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4" event={"ID":"434c1fe0-78fb-411c-a3dc-cff8157ccd08","Type":"ContainerDied","Data":"95a705b57112f98e2b7a64f5c45a5c2efc355c55e021d9a4054a22aba56acaf9"}
Sep 11 22:09:38.254447 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.254431    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4" event={"ID":"434c1fe0-78fb-411c-a3dc-cff8157ccd08","Type":"ContainerDied","Data":"c68a205caf31fe5375a3a925d6c03c3d5d37833637a543e2987472573b1daf39"}
Sep 11 22:09:38.254482 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.254459    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c68a205caf31fe5375a3a925d6c03c3d5d37833637a543e2987472573b1daf39"
Sep 11 22:09:38.255710 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.255680    2574 generic.go:356] "Generic (PLEG): container finished" podID="a763b94b-084a-4543-806e-6ac72a75e2aa" containerID="64f5e04a1b0fd8086dc5ad846a4a52d5200c55f2858b940cbd7f31445f892970" exitCode=0
Sep 11 22:09:38.255812 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.255758    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb" event={"ID":"a763b94b-084a-4543-806e-6ac72a75e2aa","Type":"ContainerDied","Data":"64f5e04a1b0fd8086dc5ad846a4a52d5200c55f2858b940cbd7f31445f892970"}
Sep 11 22:09:38.255856 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.255814    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb" event={"ID":"a763b94b-084a-4543-806e-6ac72a75e2aa","Type":"ContainerDied","Data":"eb47064bf01b255d83469bb59f693d54535cc0d52d5127682780679ec97c8142"}
Sep 11 22:09:38.255856 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.255838    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb47064bf01b255d83469bb59f693d54535cc0d52d5127682780679ec97c8142"
Sep 11 22:09:38.269897 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.269874    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:09:38.273850 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.273823    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"
Sep 11 22:09:38.276471 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.276449    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-maas-api-tls\") pod \"a763b94b-084a-4543-806e-6ac72a75e2aa\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") "
Sep 11 22:09:38.276582 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.276487    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-metrics-certs\") pod \"a763b94b-084a-4543-806e-6ac72a75e2aa\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") "
Sep 11 22:09:38.276647 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.276602    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfmnb\" (UniqueName: \"kubernetes.io/projected/a763b94b-084a-4543-806e-6ac72a75e2aa-kube-api-access-cfmnb\") pod \"a763b94b-084a-4543-806e-6ac72a75e2aa\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") "
Sep 11 22:09:38.276647 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.276636    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434c1fe0-78fb-411c-a3dc-cff8157ccd08-plugins-config-volume\") pod \"434c1fe0-78fb-411c-a3dc-cff8157ccd08\" (UID: \"434c1fe0-78fb-411c-a3dc-cff8157ccd08\") "
Sep 11 22:09:38.276752 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.276668    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a763b94b-084a-4543-806e-6ac72a75e2aa-openshift-service-ca\") pod \"a763b94b-084a-4543-806e-6ac72a75e2aa\" (UID: \"a763b94b-084a-4543-806e-6ac72a75e2aa\") "
Sep 11 22:09:38.276752 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.276720    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc87t\" (UniqueName: \"kubernetes.io/projected/434c1fe0-78fb-411c-a3dc-cff8157ccd08-kube-api-access-sc87t\") pod \"434c1fe0-78fb-411c-a3dc-cff8157ccd08\" (UID: \"434c1fe0-78fb-411c-a3dc-cff8157ccd08\") "
Sep 11 22:09:38.277097 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.277048    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434c1fe0-78fb-411c-a3dc-cff8157ccd08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "434c1fe0-78fb-411c-a3dc-cff8157ccd08" (UID: "434c1fe0-78fb-411c-a3dc-cff8157ccd08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:09:38.277500 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.277474    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a763b94b-084a-4543-806e-6ac72a75e2aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a763b94b-084a-4543-806e-6ac72a75e2aa" (UID: "a763b94b-084a-4543-806e-6ac72a75e2aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:09:38.279021 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.278972    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a763b94b-084a-4543-806e-6ac72a75e2aa" (UID: "a763b94b-084a-4543-806e-6ac72a75e2aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:09:38.279021 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.278983    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a763b94b-084a-4543-806e-6ac72a75e2aa" (UID: "a763b94b-084a-4543-806e-6ac72a75e2aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:09:38.279161 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.279140    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434c1fe0-78fb-411c-a3dc-cff8157ccd08-kube-api-access-sc87t" (OuterVolumeSpecName: "kube-api-access-sc87t") pod "434c1fe0-78fb-411c-a3dc-cff8157ccd08" (UID: "434c1fe0-78fb-411c-a3dc-cff8157ccd08"). InnerVolumeSpecName "kube-api-access-sc87t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:09:38.279752 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.279735    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a763b94b-084a-4543-806e-6ac72a75e2aa-kube-api-access-cfmnb" (OuterVolumeSpecName: "kube-api-access-cfmnb") pod "a763b94b-084a-4543-806e-6ac72a75e2aa" (UID: "a763b94b-084a-4543-806e-6ac72a75e2aa"). InnerVolumeSpecName "kube-api-access-cfmnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:09:38.377857 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.377825    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfmnb\" (UniqueName: \"kubernetes.io/projected/a763b94b-084a-4543-806e-6ac72a75e2aa-kube-api-access-cfmnb\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:09:38.377857 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.377852    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434c1fe0-78fb-411c-a3dc-cff8157ccd08-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:09:38.377857 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.377861    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a763b94b-084a-4543-806e-6ac72a75e2aa-openshift-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:09:38.378289 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.377870    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc87t\" (UniqueName: \"kubernetes.io/projected/434c1fe0-78fb-411c-a3dc-cff8157ccd08-kube-api-access-sc87t\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:09:38.378289 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.377880    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-maas-api-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:09:38.378289 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.377888    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a763b94b-084a-4543-806e-6ac72a75e2aa-metrics-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:09:38.477253 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.477171    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:38.578322 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:38.578270    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:39.261766 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:39.261737    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"
Sep 11 22:09:39.261936 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:39.261737    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"
Sep 11 22:09:39.274699 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:39.274673    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:39.286401 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:39.286377    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"]
Sep 11 22:09:39.291865 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:39.291846    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0aea78-84466784cf-mnzqb"]
Sep 11 22:09:39.302771 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:39.302748    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"]
Sep 11 22:09:39.308431 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:39.308411    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0aea78-69d7c5bc7-tjqm4"]
Sep 11 22:09:39.521918 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:39.521821    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:39.622085 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:39.622042    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:39.722453 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:39.722420    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="434c1fe0-78fb-411c-a3dc-cff8157ccd08" path="/var/lib/kubelet/pods/434c1fe0-78fb-411c-a3dc-cff8157ccd08/volumes"
Sep 11 22:09:39.723052 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:39.723024    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a763b94b-084a-4543-806e-6ac72a75e2aa" path="/var/lib/kubelet/pods/a763b94b-084a-4543-806e-6ac72a75e2aa/volumes"
Sep 11 22:09:39.788177 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:39.788093    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:40.008562 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:40.008525    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:40.110143 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:40.110106    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:40.208578 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:40.208545    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:40.308300 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:40.308266    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:41.109148 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:41.109110    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:41.914765 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:41.914731    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:43.583203 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:43.583147    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:44.323033 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:44.322992    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:44.379942 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:44.379905    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:44.653037 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:44.653003    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:45.008004 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:45.007915    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:45.326908 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:45.326823    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:45.397992 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:45.397958    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:46.102271 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:46.102235    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:46.205104 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:46.205066    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:46.302515 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:46.302486    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:46.486240 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:46.486156    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:47.104859 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:47.104820    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:47.212211 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:47.212177    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:47.303435 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:47.303400    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:48.268244 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:48.268208    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:49.844457 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:49.844423    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:52.097155 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:52.097115    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:52.155781 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:52.155750    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:52.998769 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:52.998733    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:53.380888 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:53.380855    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:53.587877 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:53.587838    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:56.518277 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:56.518236    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:56.846545 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:56.846511    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:56.941409 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:56.941370    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:57.541718 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:57.541683    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:58.318653 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:58.318614    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:59.227530 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.227501    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:59.900915 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.900883    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"]
Sep 11 22:09:59.901416 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.901397    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9248058a-db49-48d5-bc59-e73532eb22e3" containerName="payload-pre-processing"
Sep 11 22:09:59.901511 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.901417    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9248058a-db49-48d5-bc59-e73532eb22e3" containerName="payload-pre-processing"
Sep 11 22:09:59.901511 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.901431    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a763b94b-084a-4543-806e-6ac72a75e2aa" containerName="maas-api"
Sep 11 22:09:59.901511 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.901439    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a763b94b-084a-4543-806e-6ac72a75e2aa" containerName="maas-api"
Sep 11 22:09:59.901511 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.901446    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326b55e2-0d6c-418a-aa3e-d2c74e968ee4" containerName="payload-processing"
Sep 11 22:09:59.901511 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.901455    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="326b55e2-0d6c-418a-aa3e-d2c74e968ee4" containerName="payload-processing"
Sep 11 22:09:59.901511 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.901468    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="434c1fe0-78fb-411c-a3dc-cff8157ccd08" containerName="payload-processing"
Sep 11 22:09:59.901511 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.901478    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="434c1fe0-78fb-411c-a3dc-cff8157ccd08" containerName="payload-processing"
Sep 11 22:09:59.901882 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.901574    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9248058a-db49-48d5-bc59-e73532eb22e3" containerName="payload-pre-processing"
Sep 11 22:09:59.901882 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.901590    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a763b94b-084a-4543-806e-6ac72a75e2aa" containerName="maas-api"
Sep 11 22:09:59.901882 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.901601    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="326b55e2-0d6c-418a-aa3e-d2c74e968ee4" containerName="payload-processing"
Sep 11 22:09:59.901882 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.901619    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="434c1fe0-78fb-411c-a3dc-cff8157ccd08" containerName="payload-processing"
Sep 11 22:09:59.904591 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.904570    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:09:59.907512 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.907486    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-5ef59c-serving-cert\""
Sep 11 22:09:59.907933 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.907908    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mc9f9\""
Sep 11 22:09:59.909004 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.908974    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 22:09:59.909004 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.908979    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 22:09:59.909177 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.909020    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:09:59.914298 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.914134    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"]
Sep 11 22:09:59.933412 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.933376    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:59.968581 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.968554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:09:59.968698 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.968609    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:09:59.968773 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.968719    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2jkx\" (UniqueName: \"kubernetes.io/projected/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-kube-api-access-s2jkx\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:09:59.968773 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:09:59.968749    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:00.034963 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.034930    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:00.070166 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.070136    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2jkx\" (UniqueName: \"kubernetes.io/projected/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-kube-api-access-s2jkx\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:00.070166 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.070169    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:00.070371 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.070218    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:00.070371 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.070258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:00.070371 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:10:00.070323    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-serving-cert: secret "maas-api-e2e-shared-a-w2-5ef59c-serving-cert" not found
Sep 11 22:10:00.070488 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:10:00.070387    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-maas-api-tls podName:e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0 nodeName:}" failed. No retries permitted until 2026-09-11 22:10:00.570367635 +0000 UTC m=+1067.305478973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-maas-api-tls") pod "maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg" (UID: "e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0") : secret "maas-api-e2e-shared-a-w2-5ef59c-serving-cert" not found
Sep 11 22:10:00.070950 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.070928    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:00.072774 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.072752    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:00.078762 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.078740    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2jkx\" (UniqueName: \"kubernetes.io/projected/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-kube-api-access-s2jkx\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:00.383401 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.383366    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:00.483430 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.483391    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:00.490650 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.490621    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"]
Sep 11 22:10:00.494483 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.494462    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"
Sep 11 22:10:00.497443 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.497424    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-5ef59c-dockercfg-s8262\""
Sep 11 22:10:00.497577 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.497504    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5ef59c\""
Sep 11 22:10:00.505264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.505242    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"]
Sep 11 22:10:00.521835 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.521805    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"]
Sep 11 22:10:00.525509 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.525490    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"
Sep 11 22:10:00.534907 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.534886    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"]
Sep 11 22:10:00.575570 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.575534    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn2f2\" (UniqueName: \"kubernetes.io/projected/b7185e0d-5978-4e82-83d1-ced33de8c4a4-kube-api-access-zn2f2\") pod \"payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4\" (UID: \"b7185e0d-5978-4e82-83d1-ced33de8c4a4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"
Sep 11 22:10:00.575730 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.575590    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:00.575730 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.575645    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh\" (UID: \"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"
Sep 11 22:10:00.575730 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:10:00.575682    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-serving-cert: secret "maas-api-e2e-shared-a-w2-5ef59c-serving-cert" not found
Sep 11 22:10:00.575730 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.575708    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7185e0d-5978-4e82-83d1-ced33de8c4a4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4\" (UID: \"b7185e0d-5978-4e82-83d1-ced33de8c4a4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"
Sep 11 22:10:00.575957 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:10:00.575758    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-maas-api-tls podName:e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0 nodeName:}" failed. No retries permitted until 2026-09-11 22:10:01.575737097 +0000 UTC m=+1068.310848426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-maas-api-tls") pod "maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg" (UID: "e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0") : secret "maas-api-e2e-shared-a-w2-5ef59c-serving-cert" not found
Sep 11 22:10:00.575957 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.575854    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h7vl\" (UniqueName: \"kubernetes.io/projected/26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c-kube-api-access-2h7vl\") pod \"payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh\" (UID: \"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"
Sep 11 22:10:00.676774 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.676684    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn2f2\" (UniqueName: \"kubernetes.io/projected/b7185e0d-5978-4e82-83d1-ced33de8c4a4-kube-api-access-zn2f2\") pod \"payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4\" (UID: \"b7185e0d-5978-4e82-83d1-ced33de8c4a4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"
Sep 11 22:10:00.676774 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.676742    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh\" (UID: \"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"
Sep 11 22:10:00.676774 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.676777    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7185e0d-5978-4e82-83d1-ced33de8c4a4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4\" (UID: \"b7185e0d-5978-4e82-83d1-ced33de8c4a4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"
Sep 11 22:10:00.677055 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.676811    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h7vl\" (UniqueName: \"kubernetes.io/projected/26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c-kube-api-access-2h7vl\") pod \"payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh\" (UID: \"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"
Sep 11 22:10:00.677525 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.677502    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh\" (UID: \"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"
Sep 11 22:10:00.677673 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.677542    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7185e0d-5978-4e82-83d1-ced33de8c4a4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4\" (UID: \"b7185e0d-5978-4e82-83d1-ced33de8c4a4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"
Sep 11 22:10:00.686783 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.686754    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn2f2\" (UniqueName: \"kubernetes.io/projected/b7185e0d-5978-4e82-83d1-ced33de8c4a4-kube-api-access-zn2f2\") pod \"payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4\" (UID: \"b7185e0d-5978-4e82-83d1-ced33de8c4a4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"
Sep 11 22:10:00.686873 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.686852    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h7vl\" (UniqueName: \"kubernetes.io/projected/26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c-kube-api-access-2h7vl\") pod \"payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh\" (UID: \"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"
Sep 11 22:10:00.768648 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.768621    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:00.806638 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.806615    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"
Sep 11 22:10:00.836229 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:00.836205    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"
Sep 11 22:10:01.080446 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:01.080417    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"]
Sep 11 22:10:01.082553 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:10:01.082527    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26851f6d_0dd2_4dbc_a9a8_9bb62fcf7b0c.slice/crio-46e074943958bc4c5808015814c481f39e6d0801d41c4267799c52c1cc59f0ed WatchSource:0}: Error finding container 46e074943958bc4c5808015814c481f39e6d0801d41c4267799c52c1cc59f0ed: Status 404 returned error can't find the container with id 46e074943958bc4c5808015814c481f39e6d0801d41c4267799c52c1cc59f0ed
Sep 11 22:10:01.125757 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:01.125730    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"]
Sep 11 22:10:01.127071 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:10:01.127030    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7185e0d_5978_4e82_83d1_ced33de8c4a4.slice/crio-ae4f80e7122adbcaccb74c08ad990c0757aea2ad7d0beff49ca488c9238a1baa WatchSource:0}: Error finding container ae4f80e7122adbcaccb74c08ad990c0757aea2ad7d0beff49ca488c9238a1baa: Status 404 returned error can't find the container with id ae4f80e7122adbcaccb74c08ad990c0757aea2ad7d0beff49ca488c9238a1baa
Sep 11 22:10:01.381248 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:01.381212    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4" event={"ID":"b7185e0d-5978-4e82-83d1-ced33de8c4a4","Type":"ContainerStarted","Data":"63c9b8b41e905d0dafcfcd40227ea5036d3255cedc20733db2a1c2d6a110e888"}
Sep 11 22:10:01.381457 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:01.381279    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"
Sep 11 22:10:01.381457 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:01.381305    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4" event={"ID":"b7185e0d-5978-4e82-83d1-ced33de8c4a4","Type":"ContainerStarted","Data":"ae4f80e7122adbcaccb74c08ad990c0757aea2ad7d0beff49ca488c9238a1baa"}
Sep 11 22:10:01.382828 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:01.382802    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh" event={"ID":"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c","Type":"ContainerStarted","Data":"576af2da08e39daa04c63b8e2b7dacb16c0cb9f1e348198325a56c26a3b380d4"}
Sep 11 22:10:01.382915 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:01.382853    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh" event={"ID":"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c","Type":"ContainerStarted","Data":"46e074943958bc4c5808015814c481f39e6d0801d41c4267799c52c1cc59f0ed"}
Sep 11 22:10:01.382915 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:01.382883    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"
Sep 11 22:10:01.405606 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:01.400394    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4" podStartSLOduration=1.400377112 podStartE2EDuration="1.400377112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:01.397062951 +0000 UTC m=+1068.132174298" watchObservedRunningTime="2026-09-11 22:10:01.400377112 +0000 UTC m=+1068.135488459"
Sep 11 22:10:01.419978 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:01.419927    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh" podStartSLOduration=1.419911466 podStartE2EDuration="1.419911466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:01.417332787 +0000 UTC m=+1068.152444131" watchObservedRunningTime="2026-09-11 22:10:01.419911466 +0000 UTC m=+1068.155022813"
Sep 11 22:10:01.586497 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:01.586459    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:01.588805 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:01.588781    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:01.719023 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:01.718931    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:02.027041 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:02.027017    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"]
Sep 11 22:10:02.028918 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:10:02.028892    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68c80ba_cdf2_43c1_8b91_9cc7c9b9afe0.slice/crio-9b4cbed932c719c87c0befef07d46cfd625e0e71ece71a23f8cd322f6750c6d0 WatchSource:0}: Error finding container 9b4cbed932c719c87c0befef07d46cfd625e0e71ece71a23f8cd322f6750c6d0: Status 404 returned error can't find the container with id 9b4cbed932c719c87c0befef07d46cfd625e0e71ece71a23f8cd322f6750c6d0
Sep 11 22:10:02.389296 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:02.389254    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg" event={"ID":"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0","Type":"ContainerStarted","Data":"9b4cbed932c719c87c0befef07d46cfd625e0e71ece71a23f8cd322f6750c6d0"}
Sep 11 22:10:03.395690 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:03.395653    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg" event={"ID":"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0","Type":"ContainerStarted","Data":"ba05c855adb2392864dc792432d784b73b2e701e72beeb010ca2a63997fe120e"}
Sep 11 22:10:03.396154 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:03.395724    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:03.414290 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:03.414232    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg" podStartSLOduration=3.909352359 podStartE2EDuration="4.41421167s" totalImagesPullingTime="504.859311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:03.412524924 +0000 UTC m=+1070.147636270" watchObservedRunningTime="2026-09-11 22:10:03.41421167 +0000 UTC m=+1070.149323017"
Sep 11 22:10:09.407391 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:09.407304    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:10:12.391718 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:12.391680    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"
Sep 11 22:10:12.392101 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:12.391757    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"
Sep 11 22:10:21.725046 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:21.725009    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:21.904818 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:21.904779    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:22.049686 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:22.049588    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:22.260236 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:22.260200    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:22.331428 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:22.331346    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:22.389362 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:22.389320    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:22.855756 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:22.855710    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:23.147585 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:23.147499    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:23.771451 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:23.771412    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:24.618996 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:24.618940    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:25.517604 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:25.517568    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:25.973093 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:25.973056    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:26.071560 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:26.071520    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:26.566716 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:26.566684    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:27.301437 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.301405    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"]
Sep 11 22:10:27.307005 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.306981    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:27.310134 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.310115    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-38f772-serving-cert\""
Sep 11 22:10:27.318618 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.318591    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"]
Sep 11 22:10:27.321572 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.321272    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qxd7\" (UniqueName: \"kubernetes.io/projected/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-kube-api-access-6qxd7\") pod \"maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:27.321572 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.321387    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:27.321572 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.321427    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:27.321572 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.321454    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:27.337484 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.337459    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:27.422093 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.422065    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:27.422272 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.422102    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:27.422272 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.422121    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:27.422272 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.422179    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qxd7\" (UniqueName: \"kubernetes.io/projected/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-kube-api-access-6qxd7\") pod \"maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:27.422272 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:10:27.422236    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-serving-cert: secret "maas-api-e2e-shared-b-w2-38f772-serving-cert" not found
Sep 11 22:10:27.422478 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:10:27.422343    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-maas-api-tls podName:5fadb3d5-8da2-4dde-8dd2-0472a4c073f9 nodeName:}" failed. No retries permitted until 2026-09-11 22:10:27.922319834 +0000 UTC m=+1094.657431173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-maas-api-tls") pod "maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc" (UID: "5fadb3d5-8da2-4dde-8dd2-0472a4c073f9") : secret "maas-api-e2e-shared-b-w2-38f772-serving-cert" not found
Sep 11 22:10:27.422891 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.422874    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:27.424704 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.424646    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:27.431286 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.431262    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qxd7\" (UniqueName: \"kubernetes.io/projected/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-kube-api-access-6qxd7\") pod \"maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:27.438086 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.438060    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:27.917849 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.917808    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:27.926211 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.926181    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:27.928536 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:27.928519    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:28.017810 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.017776    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:28.033951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.033918    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"]
Sep 11 22:10:28.037899 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.037876    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"
Sep 11 22:10:28.040997 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.040975    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-38f772-dockercfg-dkrqg\""
Sep 11 22:10:28.041115 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.041018    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-38f772\""
Sep 11 22:10:28.060862 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.060830    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"]
Sep 11 22:10:28.127323 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.127288    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c696561-1418-46fe-9f78-4c53b3544548-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6\" (UID: \"2c696561-1418-46fe-9f78-4c53b3544548\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"
Sep 11 22:10:28.127522 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.127336    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm99x\" (UniqueName: \"kubernetes.io/projected/2c696561-1418-46fe-9f78-4c53b3544548-kube-api-access-gm99x\") pod \"payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6\" (UID: \"2c696561-1418-46fe-9f78-4c53b3544548\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"
Sep 11 22:10:28.223022 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.222945    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:28.228600 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.228547    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c696561-1418-46fe-9f78-4c53b3544548-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6\" (UID: \"2c696561-1418-46fe-9f78-4c53b3544548\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"
Sep 11 22:10:28.228600 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.228587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm99x\" (UniqueName: \"kubernetes.io/projected/2c696561-1418-46fe-9f78-4c53b3544548-kube-api-access-gm99x\") pod \"payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6\" (UID: \"2c696561-1418-46fe-9f78-4c53b3544548\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"
Sep 11 22:10:28.229254 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.229226    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c696561-1418-46fe-9f78-4c53b3544548-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6\" (UID: \"2c696561-1418-46fe-9f78-4c53b3544548\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"
Sep 11 22:10:28.239655 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.239632    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm99x\" (UniqueName: \"kubernetes.io/projected/2c696561-1418-46fe-9f78-4c53b3544548-kube-api-access-gm99x\") pod \"payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6\" (UID: \"2c696561-1418-46fe-9f78-4c53b3544548\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"
Sep 11 22:10:28.315852 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.315786    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:28.352377 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.352337    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"
Sep 11 22:10:28.495890 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.495863    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"]
Sep 11 22:10:28.497884 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:10:28.497857    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fadb3d5_8da2_4dde_8dd2_0472a4c073f9.slice/crio-dfce887381fd140c9b5ac6ca1839b2f748d0c48f766cd84485dd7ad314d38b92 WatchSource:0}: Error finding container dfce887381fd140c9b5ac6ca1839b2f748d0c48f766cd84485dd7ad314d38b92: Status 404 returned error can't find the container with id dfce887381fd140c9b5ac6ca1839b2f748d0c48f766cd84485dd7ad314d38b92
Sep 11 22:10:28.547405 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.547366    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc" event={"ID":"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9","Type":"ContainerStarted","Data":"dfce887381fd140c9b5ac6ca1839b2f748d0c48f766cd84485dd7ad314d38b92"}
Sep 11 22:10:28.634558 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:28.632499    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"]
Sep 11 22:10:28.635181 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:10:28.635135    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c696561_1418_46fe_9f78_4c53b3544548.slice/crio-9a9b7f96e32a7a487f9a05b93236514a7f59b22945a0bded889afeed33fec1a0 WatchSource:0}: Error finding container 9a9b7f96e32a7a487f9a05b93236514a7f59b22945a0bded889afeed33fec1a0: Status 404 returned error can't find the container with id 9a9b7f96e32a7a487f9a05b93236514a7f59b22945a0bded889afeed33fec1a0
Sep 11 22:10:29.148455 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:29.148419    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:29.555461 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:29.555365    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc" event={"ID":"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9","Type":"ContainerStarted","Data":"3f72d1d3db2787bf333f6be5b0dc2189e006122d52564e74332e950791472b6c"}
Sep 11 22:10:29.555969 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:29.555455    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:29.557265 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:29.557240    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6" event={"ID":"2c696561-1418-46fe-9f78-4c53b3544548","Type":"ContainerStarted","Data":"274b1c59dbf63b9cefa092941c1800a111f0c009bc997efb1c9ba9c9ce33f46b"}
Sep 11 22:10:29.557393 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:29.557285    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6" event={"ID":"2c696561-1418-46fe-9f78-4c53b3544548","Type":"ContainerStarted","Data":"9a9b7f96e32a7a487f9a05b93236514a7f59b22945a0bded889afeed33fec1a0"}
Sep 11 22:10:29.557393 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:29.557340    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"
Sep 11 22:10:29.576015 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:29.575977    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc" podStartSLOduration=2.257901416 podStartE2EDuration="2.575964882s" totalImagesPullingTime="318.063466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:29.572873038 +0000 UTC m=+1096.307984385" watchObservedRunningTime="2026-09-11 22:10:29.575964882 +0000 UTC m=+1096.311076251"
Sep 11 22:10:29.589734 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:29.589693    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6" podStartSLOduration=1.589682483 podStartE2EDuration="1.589682483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:29.588750501 +0000 UTC m=+1096.323861847" watchObservedRunningTime="2026-09-11 22:10:29.589682483 +0000 UTC m=+1096.324793837"
Sep 11 22:10:29.912264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:29.912224    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:30.711895 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:30.711834    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:31.608875 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:31.608844    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:32.679082 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:32.679045    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:34.019579 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:34.019542    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:34.897782 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:34.897741    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:35.542554 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:35.542522    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:35.568560 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:35.568534    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:10:36.469769 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:36.469734    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:37.319643 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:37.319610    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:38.033182 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:38.033147    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:38.920855 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:38.920818    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:39.625401 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:39.625365    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:40.329665 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:40.329629    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:40.564914 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:40.564888    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"
Sep 11 22:10:40.683647 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:40.683616    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:41.246352 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:41.246293    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:42.109556 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:42.109490    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:44.206623 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:44.206589    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:44.777962 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:44.777919    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:44.818152 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:44.818118    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"]
Sep 11 22:10:44.818489 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:44.818457    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn" podUID="1e375557-c5d4-4752-b35a-69dbfcfc82a2" containerName="payload-pre-processing" containerID="cri-o://f671cc5d207baf3f4f801e8dfc78b9638d85b9a44db7f2c24936dea2a9167c81" gracePeriod=30
Sep 11 22:10:44.884347 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:44.879907    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:44.979736 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:44.979699    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:45.035367 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.035270    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:45.093584 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.093559    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"
Sep 11 22:10:45.190795 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.190756    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e375557-c5d4-4752-b35a-69dbfcfc82a2-plugins-config-volume\") pod \"1e375557-c5d4-4752-b35a-69dbfcfc82a2\" (UID: \"1e375557-c5d4-4752-b35a-69dbfcfc82a2\") "
Sep 11 22:10:45.190975 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.190816    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7x5g\" (UniqueName: \"kubernetes.io/projected/1e375557-c5d4-4752-b35a-69dbfcfc82a2-kube-api-access-k7x5g\") pod \"1e375557-c5d4-4752-b35a-69dbfcfc82a2\" (UID: \"1e375557-c5d4-4752-b35a-69dbfcfc82a2\") "
Sep 11 22:10:45.191128 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.191102    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e375557-c5d4-4752-b35a-69dbfcfc82a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e375557-c5d4-4752-b35a-69dbfcfc82a2" (UID: "1e375557-c5d4-4752-b35a-69dbfcfc82a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:10:45.192920 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.192901    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e375557-c5d4-4752-b35a-69dbfcfc82a2-kube-api-access-k7x5g" (OuterVolumeSpecName: "kube-api-access-k7x5g") pod "1e375557-c5d4-4752-b35a-69dbfcfc82a2" (UID: "1e375557-c5d4-4752-b35a-69dbfcfc82a2"). InnerVolumeSpecName "kube-api-access-k7x5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:10:45.291922 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.291856    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e375557-c5d4-4752-b35a-69dbfcfc82a2-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:10:45.291922 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.291884    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7x5g\" (UniqueName: \"kubernetes.io/projected/1e375557-c5d4-4752-b35a-69dbfcfc82a2-kube-api-access-k7x5g\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:10:45.417894 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.417857    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:45.513584 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.513546    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:45.646715 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.646684    2574 generic.go:356] "Generic (PLEG): container finished" podID="1e375557-c5d4-4752-b35a-69dbfcfc82a2" containerID="f671cc5d207baf3f4f801e8dfc78b9638d85b9a44db7f2c24936dea2a9167c81" exitCode=0
Sep 11 22:10:45.646894 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.646726    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"
Sep 11 22:10:45.646894 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.646781    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn" event={"ID":"1e375557-c5d4-4752-b35a-69dbfcfc82a2","Type":"ContainerDied","Data":"f671cc5d207baf3f4f801e8dfc78b9638d85b9a44db7f2c24936dea2a9167c81"}
Sep 11 22:10:45.646894 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.646828    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn" event={"ID":"1e375557-c5d4-4752-b35a-69dbfcfc82a2","Type":"ContainerDied","Data":"34ca9a1df2a32020982b664b54003a5bae6412925e2c772d2bde67e8e2545775"}
Sep 11 22:10:45.646894 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.646851    2574 scope.go:117] "RemoveContainer" containerID="f671cc5d207baf3f4f801e8dfc78b9638d85b9a44db7f2c24936dea2a9167c81"
Sep 11 22:10:45.655846 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.655827    2574 scope.go:117] "RemoveContainer" containerID="f671cc5d207baf3f4f801e8dfc78b9638d85b9a44db7f2c24936dea2a9167c81"
Sep 11 22:10:45.656114 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:10:45.656096    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f671cc5d207baf3f4f801e8dfc78b9638d85b9a44db7f2c24936dea2a9167c81\": container with ID starting with f671cc5d207baf3f4f801e8dfc78b9638d85b9a44db7f2c24936dea2a9167c81 not found: ID does not exist" containerID="f671cc5d207baf3f4f801e8dfc78b9638d85b9a44db7f2c24936dea2a9167c81"
Sep 11 22:10:45.656167 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.656124    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f671cc5d207baf3f4f801e8dfc78b9638d85b9a44db7f2c24936dea2a9167c81"} err="failed to get container status \"f671cc5d207baf3f4f801e8dfc78b9638d85b9a44db7f2c24936dea2a9167c81\": rpc error: code = NotFound desc = could not find container \"f671cc5d207baf3f4f801e8dfc78b9638d85b9a44db7f2c24936dea2a9167c81\": container with ID starting with f671cc5d207baf3f4f801e8dfc78b9638d85b9a44db7f2c24936dea2a9167c81 not found: ID does not exist"
Sep 11 22:10:45.670697 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.670677    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"]
Sep 11 22:10:45.674991 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.674969    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27dad9-75f465f7d6-7mvfn"]
Sep 11 22:10:45.720659 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:45.720636    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e375557-c5d4-4752-b35a-69dbfcfc82a2" path="/var/lib/kubelet/pods/1e375557-c5d4-4752-b35a-69dbfcfc82a2/volumes"
Sep 11 22:10:46.203199 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:46.203146    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:46.977124 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:46.977091    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:47.709647 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:47.709612    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:48.519974 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:48.519924    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:49.422707 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:49.422668    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:51.017725 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:51.017688    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:51.075832 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:51.075800    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:51.296802 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:51.296717    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:52.506117 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:52.506081    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:52.631660 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:52.631579    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:52.817554 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:52.817517    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:53.348845 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:53.348810    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:53.676939 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:53.676849    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:54.033951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:54.033854    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:54.809204 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:54.809168    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:55.505661 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:55.505619    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:56.223743 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:56.223707    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:56.318758 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:56.318724    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:57.229033 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:57.228999    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:58.234273 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:58.234239    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:58.942907 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:58.942874    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:59.681436 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:10:59.681402    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:00.320806 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:00.320771    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:00.968421 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:00.968389    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:01.729254 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:01.729220    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:01.791121 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:01.791093    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:02.118829 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.118798    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:02.220847 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.220813    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:02.298778 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.298745    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:02.345355 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.345325    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"]
Sep 11 22:11:02.345771 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.345759    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e375557-c5d4-4752-b35a-69dbfcfc82a2" containerName="payload-pre-processing"
Sep 11 22:11:02.345816 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.345773    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e375557-c5d4-4752-b35a-69dbfcfc82a2" containerName="payload-pre-processing"
Sep 11 22:11:02.345861 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.345852    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e375557-c5d4-4752-b35a-69dbfcfc82a2" containerName="payload-pre-processing"
Sep 11 22:11:02.350614 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.350596    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"
Sep 11 22:11:02.355271 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.355251    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-92d3298e\""
Sep 11 22:11:02.355418 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.355369    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-92d3298e-dockercfg-nxwtr\""
Sep 11 22:11:02.360479 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.360456    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"]
Sep 11 22:11:02.456372 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.456268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde830ab-18b8-43f7-82c8-54f6aa447a4e-plugins-config-volume\") pod \"payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp\" (UID: \"bde830ab-18b8-43f7-82c8-54f6aa447a4e\") " pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"
Sep 11 22:11:02.456520 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.456433    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbf6h\" (UniqueName: \"kubernetes.io/projected/bde830ab-18b8-43f7-82c8-54f6aa447a4e-kube-api-access-lbf6h\") pod \"payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp\" (UID: \"bde830ab-18b8-43f7-82c8-54f6aa447a4e\") " pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"
Sep 11 22:11:02.496828 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.496791    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:02.557739 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.557707    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde830ab-18b8-43f7-82c8-54f6aa447a4e-plugins-config-volume\") pod \"payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp\" (UID: \"bde830ab-18b8-43f7-82c8-54f6aa447a4e\") " pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"
Sep 11 22:11:02.557893 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.557787    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbf6h\" (UniqueName: \"kubernetes.io/projected/bde830ab-18b8-43f7-82c8-54f6aa447a4e-kube-api-access-lbf6h\") pod \"payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp\" (UID: \"bde830ab-18b8-43f7-82c8-54f6aa447a4e\") " pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"
Sep 11 22:11:02.558637 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.558613    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde830ab-18b8-43f7-82c8-54f6aa447a4e-plugins-config-volume\") pod \"payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp\" (UID: \"bde830ab-18b8-43f7-82c8-54f6aa447a4e\") " pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"
Sep 11 22:11:02.567617 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.567591    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbf6h\" (UniqueName: \"kubernetes.io/projected/bde830ab-18b8-43f7-82c8-54f6aa447a4e-kube-api-access-lbf6h\") pod \"payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp\" (UID: \"bde830ab-18b8-43f7-82c8-54f6aa447a4e\") " pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"
Sep 11 22:11:02.567782 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.567740    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:02.647521 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.647428    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:02.661587 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.661562    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"
Sep 11 22:11:02.745382 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.745352    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:02.935356 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:02.935299    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"]
Sep 11 22:11:02.936945 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:11:02.936917    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde830ab_18b8_43f7_82c8_54f6aa447a4e.slice/crio-22d1fef8ca49ad4297233640fcfbf50d8432e033dcaa297a55030d6c6cc4bfbd WatchSource:0}: Error finding container 22d1fef8ca49ad4297233640fcfbf50d8432e033dcaa297a55030d6c6cc4bfbd: Status 404 returned error can't find the container with id 22d1fef8ca49ad4297233640fcfbf50d8432e033dcaa297a55030d6c6cc4bfbd
Sep 11 22:11:03.115417 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:03.115385    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:03.504888 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:03.504854    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:03.750126 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:03.750091    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp" event={"ID":"bde830ab-18b8-43f7-82c8-54f6aa447a4e","Type":"ContainerStarted","Data":"e5287c66c810364962ec4a4614348ce9b035e88a893db15986cd1ea8d8ca4ee8"}
Sep 11 22:11:03.750284 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:03.750156    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"
Sep 11 22:11:03.750284 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:03.750183    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp" event={"ID":"bde830ab-18b8-43f7-82c8-54f6aa447a4e","Type":"ContainerStarted","Data":"22d1fef8ca49ad4297233640fcfbf50d8432e033dcaa297a55030d6c6cc4bfbd"}
Sep 11 22:11:03.769705 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:03.769620    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp" podStartSLOduration=1.7696068710000001 podStartE2EDuration="1.769606871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:11:03.766500255 +0000 UTC m=+1130.501611601" watchObservedRunningTime="2026-09-11 22:11:03.769606871 +0000 UTC m=+1130.504718218"
Sep 11 22:11:04.317772 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:04.317732    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:05.005687 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:05.005648    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:05.799783 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:05.799745    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:06.529938 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:06.529901    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:07.288941 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:07.288902    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:08.188780 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:08.188748    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:13.826695 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:13.826664    2574 scope.go:117] "RemoveContainer" containerID="1cf10301aae63bf106539d9ad5d303c3b300ee53d1f41fe3609e91f6c16c8bdf"
Sep 11 22:11:14.758008 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:14.757979    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"
Sep 11 22:11:23.844046 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:23.844011    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:24.780058 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:24.780025    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:25.502214 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:25.502167    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:26.232496 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:26.232465    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:26.332254 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:26.332222    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:27.127182 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:27.127152    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:27.810418 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:27.810391    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:28.012426 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:28.012388    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:40.338639 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:40.338607    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:41.606266 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:41.606229    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:44.746216 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:44.746181    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:44.999709 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:44.999621    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:45.553793 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:45.553754    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:45.865131 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:45.865098    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:46.280613 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:46.280540    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:47.900286 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:47.900255    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:48.520103 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:48.520069    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:48.617531 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:48.617492    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:49.486109 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:49.486059    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:50.283776 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:50.283746    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:51.044154 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:51.044096    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:51.731403 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:51.731365    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:52.513638 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:52.513591    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:52.615957 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:52.615873    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:52.719121 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:52.719083    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:52.997086 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:52.997009    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:53.093698 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:53.093655    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:56.171386 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:11:56.171342    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:01.531955 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:01.531916    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:01.769617 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:01.769577    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:01.867163 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:01.867135    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:06.944717 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:06.944681    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:13.741907 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:13.741879    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:12:13.743077 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:13.743053    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:12:13.746066 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:13.746043    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:12:13.747201 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:13.747183    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:12:13.851040 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:13.851003    2574 scope.go:117] "RemoveContainer" containerID="941b49060e3e3528cfad9adab1b4e35bcf26052e3b80f4fa1e18203da7caa595"
Sep 11 22:12:14.713739 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:14.713700    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:15.499923 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:15.499878    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:16.179235 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:16.179196    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:16.895944 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:16.895907    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:17.723077 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:17.723038    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:18.537906 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:18.537870    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:18.636104 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:18.636053    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:18.903149 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:18.903106    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:19.211016 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:19.210927    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:19.336497 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:19.336447    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:19.793346 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:19.793305    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:21.820253 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:21.820210    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:22.441057 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:22.440910    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:22.543455 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:22.543421    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:22.663090 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:22.663056    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:23.518373 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:23.518336    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:24.192869 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:24.192832    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:24.878795 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:24.878760    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:25.600360 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:25.600300    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:26.288448 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:26.288411    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:27.152154 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:27.152120    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:27.274526 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:27.274482    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:27.382833 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:27.382802    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:27.700195 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:27.700158    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:27.769849 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:27.769788    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:43.725678 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:43.725640    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:43.854582 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:43.854536    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:49.199616 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:49.199580    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:49.977665 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:49.977616    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:50.623440 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:50.623398    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:51.280602 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:51.280565    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:52.575524 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:52.575491    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:52.800249 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:52.800168    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:53.626763 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:53.626722    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:54.294682 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:54.294648    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:59.390737 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:12:59.390690    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:13:04.483853 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:13:04.483815    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:13:09.651566 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:13:09.651483    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:13:13.373742 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:13:13.373711    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:13:13.872001 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:13:13.871970    2574 scope.go:117] "RemoveContainer" containerID="e63154adb21cebe49be7edc43793d10064386a018b65e7988cf62c117cd64e43"
Sep 11 22:13:15.598491 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:13:15.598458    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:13:15.699025 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:13:15.698995    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:13:26.204770 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:13:26.204731    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:13:34.014942 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:13:34.014905    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:01.535583 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:01.535544    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:09.706724 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:09.706686    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:10.402772 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:10.402740    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:11.206871 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:11.206838    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:11.636850 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:11.636815    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:12.000653 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:12.000563    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:12.831839 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:12.831806    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:13.491166 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:13.491126    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:13.591488 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:13.591454    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:13.644847 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:13.644815    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:13.893919 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:13.893895    2574 scope.go:117] "RemoveContainer" containerID="64f5e04a1b0fd8086dc5ad846a4a52d5200c55f2858b940cbd7f31445f892970"
Sep 11 22:14:13.902384 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:13.902364    2574 scope.go:117] "RemoveContainer" containerID="95a705b57112f98e2b7a64f5c45a5c2efc355c55e021d9a4054a22aba56acaf9"
Sep 11 22:14:14.253485 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:14.253396    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:15.014125 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:15.014082    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:15.682856 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:15.682825    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:22.277062 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:22.277024    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:24.884549 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:24.884509    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:24.915431 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:24.915390    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"]
Sep 11 22:14:24.916423 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:24.916129    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp" podUID="bde830ab-18b8-43f7-82c8-54f6aa447a4e" containerName="payload-processing" containerID="cri-o://e5287c66c810364962ec4a4614348ce9b035e88a893db15986cd1ea8d8ca4ee8" gracePeriod=30
Sep 11 22:14:24.950741 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:24.950556    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:25.014513 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.014475    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:25.090088 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.089873    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:25.158860 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.156347    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:25.177793 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.177768    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"
Sep 11 22:14:25.283193 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.283160    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbf6h\" (UniqueName: \"kubernetes.io/projected/bde830ab-18b8-43f7-82c8-54f6aa447a4e-kube-api-access-lbf6h\") pod \"bde830ab-18b8-43f7-82c8-54f6aa447a4e\" (UID: \"bde830ab-18b8-43f7-82c8-54f6aa447a4e\") "
Sep 11 22:14:25.283372 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.283216    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde830ab-18b8-43f7-82c8-54f6aa447a4e-plugins-config-volume\") pod \"bde830ab-18b8-43f7-82c8-54f6aa447a4e\" (UID: \"bde830ab-18b8-43f7-82c8-54f6aa447a4e\") "
Sep 11 22:14:25.283605 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.283583    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde830ab-18b8-43f7-82c8-54f6aa447a4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bde830ab-18b8-43f7-82c8-54f6aa447a4e" (UID: "bde830ab-18b8-43f7-82c8-54f6aa447a4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:14:25.285218 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.285199    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde830ab-18b8-43f7-82c8-54f6aa447a4e-kube-api-access-lbf6h" (OuterVolumeSpecName: "kube-api-access-lbf6h") pod "bde830ab-18b8-43f7-82c8-54f6aa447a4e" (UID: "bde830ab-18b8-43f7-82c8-54f6aa447a4e"). InnerVolumeSpecName "kube-api-access-lbf6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:14:25.384256 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.384227    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbf6h\" (UniqueName: \"kubernetes.io/projected/bde830ab-18b8-43f7-82c8-54f6aa447a4e-kube-api-access-lbf6h\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:14:25.384256 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.384250    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde830ab-18b8-43f7-82c8-54f6aa447a4e-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:14:25.522536 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.522451    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:25.872561 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.872528    2574 generic.go:356] "Generic (PLEG): container finished" podID="bde830ab-18b8-43f7-82c8-54f6aa447a4e" containerID="e5287c66c810364962ec4a4614348ce9b035e88a893db15986cd1ea8d8ca4ee8" exitCode=0
Sep 11 22:14:25.872561 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.872567    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp" event={"ID":"bde830ab-18b8-43f7-82c8-54f6aa447a4e","Type":"ContainerDied","Data":"e5287c66c810364962ec4a4614348ce9b035e88a893db15986cd1ea8d8ca4ee8"}
Sep 11 22:14:25.872818 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.872593    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"
Sep 11 22:14:25.872818 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.872607    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp" event={"ID":"bde830ab-18b8-43f7-82c8-54f6aa447a4e","Type":"ContainerDied","Data":"22d1fef8ca49ad4297233640fcfbf50d8432e033dcaa297a55030d6c6cc4bfbd"}
Sep 11 22:14:25.872818 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.872628    2574 scope.go:117] "RemoveContainer" containerID="e5287c66c810364962ec4a4614348ce9b035e88a893db15986cd1ea8d8ca4ee8"
Sep 11 22:14:25.881691 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.881672    2574 scope.go:117] "RemoveContainer" containerID="e5287c66c810364962ec4a4614348ce9b035e88a893db15986cd1ea8d8ca4ee8"
Sep 11 22:14:25.882007 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:14:25.881989    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5287c66c810364962ec4a4614348ce9b035e88a893db15986cd1ea8d8ca4ee8\": container with ID starting with e5287c66c810364962ec4a4614348ce9b035e88a893db15986cd1ea8d8ca4ee8 not found: ID does not exist" containerID="e5287c66c810364962ec4a4614348ce9b035e88a893db15986cd1ea8d8ca4ee8"
Sep 11 22:14:25.882065 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.882013    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5287c66c810364962ec4a4614348ce9b035e88a893db15986cd1ea8d8ca4ee8"} err="failed to get container status \"e5287c66c810364962ec4a4614348ce9b035e88a893db15986cd1ea8d8ca4ee8\": rpc error: code = NotFound desc = could not find container \"e5287c66c810364962ec4a4614348ce9b035e88a893db15986cd1ea8d8ca4ee8\": container with ID starting with e5287c66c810364962ec4a4614348ce9b035e88a893db15986cd1ea8d8ca4ee8 not found: ID does not exist"
Sep 11 22:14:25.889381 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.889351    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"]
Sep 11 22:14:25.894582 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:25.894560    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-92d3298e-5c5cbd657c-sljhp"]
Sep 11 22:14:26.214065 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:26.213975    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:27.056469 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:27.056435    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:27.720570 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:27.720518    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:27.722052 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:27.722031    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bde830ab-18b8-43f7-82c8-54f6aa447a4e" path="/var/lib/kubelet/pods/bde830ab-18b8-43f7-82c8-54f6aa447a4e/volumes"
Sep 11 22:14:28.208719 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:28.208687    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:28.543375 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:28.543268    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:29.308132 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:29.308100    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:29.974453 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:29.974395    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:33.005842 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:33.005807    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:14:33.238486 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:14:33.238442    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:00.160527 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.160490    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c"]
Sep 11 22:15:00.161002 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.160953    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bde830ab-18b8-43f7-82c8-54f6aa447a4e" containerName="payload-processing"
Sep 11 22:15:00.161002 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.160968    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bde830ab-18b8-43f7-82c8-54f6aa447a4e" containerName="payload-processing"
Sep 11 22:15:00.161076 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.161063    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bde830ab-18b8-43f7-82c8-54f6aa447a4e" containerName="payload-processing"
Sep 11 22:15:00.164206 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.164188    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819415-wrdw9"]
Sep 11 22:15:00.164318 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.164293    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c"
Sep 11 22:15:00.167408 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.167390    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819415-wrdw9"
Sep 11 22:15:00.167554 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.167540    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:15:00.175234 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.175214    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c"
Sep 11 22:15:00.181619 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.181604    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819415-wrdw9"
Sep 11 22:15:00.189883 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.189862    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c"]
Sep 11 22:15:00.191694 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.191668    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819415-wrdw9"]
Sep 11 22:15:00.198204 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.198184    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75"]
Sep 11 22:15:00.202696 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.202672    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75"
Sep 11 22:15:00.220544 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.220507    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75"]
Sep 11 22:15:00.234123 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.234087    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75"
Sep 11 22:15:00.477249 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.477223    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c"]
Sep 11 22:15:00.479245 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:15:00.479219    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ae0db7_4d02_4488_be6c_b4ea556f9239.slice/crio-11f385e59f9f54474a5b24de0303f2306d60cbdebf702b0dc8f2d5f7899ca074 WatchSource:0}: Error finding container 11f385e59f9f54474a5b24de0303f2306d60cbdebf702b0dc8f2d5f7899ca074: Status 404 returned error can't find the container with id 11f385e59f9f54474a5b24de0303f2306d60cbdebf702b0dc8f2d5f7899ca074
Sep 11 22:15:00.494409 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.494387    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819415-wrdw9"]
Sep 11 22:15:00.496707 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:15:00.496684    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b28c946_2d0d_4188_8858_cdaf94fa2353.slice/crio-c8ee7c9f51b73d35817b1ab3111dbe34781e256b3fbb8359d59bf8f65d9d2f81 WatchSource:0}: Error finding container c8ee7c9f51b73d35817b1ab3111dbe34781e256b3fbb8359d59bf8f65d9d2f81: Status 404 returned error can't find the container with id c8ee7c9f51b73d35817b1ab3111dbe34781e256b3fbb8359d59bf8f65d9d2f81
Sep 11 22:15:00.536343 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:00.536322    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75"]
Sep 11 22:15:00.538004 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:15:00.537980    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2110600_2cd7_440c_8b8f_ea6aece3be2b.slice/crio-d87216af1786d31f63f788a321c7667ed828b154f234b14b91d08b1405a22a2c WatchSource:0}: Error finding container d87216af1786d31f63f788a321c7667ed828b154f234b14b91d08b1405a22a2c: Status 404 returned error can't find the container with id d87216af1786d31f63f788a321c7667ed828b154f234b14b91d08b1405a22a2c
Sep 11 22:15:01.069920 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:01.069839    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75" event={"ID":"c2110600-2cd7-440c-8b8f-ea6aece3be2b","Type":"ContainerStarted","Data":"688244c6b5672a5cbd0621f05768e191d6ec532735a7fcc0c4f7cfe65d9e9122"}
Sep 11 22:15:01.069920 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:01.069903    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75" event={"ID":"c2110600-2cd7-440c-8b8f-ea6aece3be2b","Type":"ContainerStarted","Data":"d87216af1786d31f63f788a321c7667ed828b154f234b14b91d08b1405a22a2c"}
Sep 11 22:15:01.071415 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:01.071386    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819415-wrdw9" event={"ID":"0b28c946-2d0d-4188-8858-cdaf94fa2353","Type":"ContainerStarted","Data":"cee77773f8772f934ef817c768dba7a7332b26dd403a55683fabd94de2b7ee2f"}
Sep 11 22:15:01.071538 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:01.071434    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819415-wrdw9" event={"ID":"0b28c946-2d0d-4188-8858-cdaf94fa2353","Type":"ContainerStarted","Data":"c8ee7c9f51b73d35817b1ab3111dbe34781e256b3fbb8359d59bf8f65d9d2f81"}
Sep 11 22:15:01.072908 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:01.072887    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c" event={"ID":"a5ae0db7-4d02-4488-be6c-b4ea556f9239","Type":"ContainerStarted","Data":"1d64b1fce00e261c2c29094677ad1fbe33263a7703da59a19b25820d2b7fbf0f"}
Sep 11 22:15:01.073002 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:01.072924    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c" event={"ID":"a5ae0db7-4d02-4488-be6c-b4ea556f9239","Type":"ContainerStarted","Data":"11f385e59f9f54474a5b24de0303f2306d60cbdebf702b0dc8f2d5f7899ca074"}
Sep 11 22:15:01.092412 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:01.092365    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75" podStartSLOduration=1.092351421 podStartE2EDuration="1.092351421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:15:01.091344114 +0000 UTC m=+1367.826455459" watchObservedRunningTime="2026-09-11 22:15:01.092351421 +0000 UTC m=+1367.827462761"
Sep 11 22:15:01.114108 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:01.114059    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c" podStartSLOduration=1.114043028 podStartE2EDuration="1.114043028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:15:01.113901288 +0000 UTC m=+1367.849012635" watchObservedRunningTime="2026-09-11 22:15:01.114043028 +0000 UTC m=+1367.849154377"
Sep 11 22:15:01.134736 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:01.134684    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819415-wrdw9" podStartSLOduration=1.134669761 podStartE2EDuration="1.134669761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:15:01.133759436 +0000 UTC m=+1367.868870782" watchObservedRunningTime="2026-09-11 22:15:01.134669761 +0000 UTC m=+1367.869781107"
Sep 11 22:15:02.080366 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:02.080303    2574 generic.go:356] "Generic (PLEG): container finished" podID="a5ae0db7-4d02-4488-be6c-b4ea556f9239" containerID="1d64b1fce00e261c2c29094677ad1fbe33263a7703da59a19b25820d2b7fbf0f" exitCode=0
Sep 11 22:15:02.080714 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:02.080340    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c" event={"ID":"a5ae0db7-4d02-4488-be6c-b4ea556f9239","Type":"ContainerDied","Data":"1d64b1fce00e261c2c29094677ad1fbe33263a7703da59a19b25820d2b7fbf0f"}
Sep 11 22:15:02.082024 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:02.081996    2574 generic.go:356] "Generic (PLEG): container finished" podID="c2110600-2cd7-440c-8b8f-ea6aece3be2b" containerID="688244c6b5672a5cbd0621f05768e191d6ec532735a7fcc0c4f7cfe65d9e9122" exitCode=0
Sep 11 22:15:02.082149 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:02.082068    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75" event={"ID":"c2110600-2cd7-440c-8b8f-ea6aece3be2b","Type":"ContainerDied","Data":"688244c6b5672a5cbd0621f05768e191d6ec532735a7fcc0c4f7cfe65d9e9122"}
Sep 11 22:15:02.083543 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:02.083516    2574 generic.go:356] "Generic (PLEG): container finished" podID="0b28c946-2d0d-4188-8858-cdaf94fa2353" containerID="cee77773f8772f934ef817c768dba7a7332b26dd403a55683fabd94de2b7ee2f" exitCode=0
Sep 11 22:15:02.083647 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:02.083551    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819415-wrdw9" event={"ID":"0b28c946-2d0d-4188-8858-cdaf94fa2353","Type":"ContainerDied","Data":"cee77773f8772f934ef817c768dba7a7332b26dd403a55683fabd94de2b7ee2f"}
Sep 11 22:15:03.245179 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:03.245154    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819415-wrdw9"
Sep 11 22:15:03.286694 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:03.286667    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75"
Sep 11 22:15:03.289928 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:03.289907    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c"
Sep 11 22:15:04.096193 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:04.096169    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c"
Sep 11 22:15:04.096408 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:04.096207    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c" event={"ID":"a5ae0db7-4d02-4488-be6c-b4ea556f9239","Type":"ContainerDied","Data":"11f385e59f9f54474a5b24de0303f2306d60cbdebf702b0dc8f2d5f7899ca074"}
Sep 11 22:15:04.096408 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:04.096265    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11f385e59f9f54474a5b24de0303f2306d60cbdebf702b0dc8f2d5f7899ca074"
Sep 11 22:15:04.097701 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:04.097678    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75" event={"ID":"c2110600-2cd7-440c-8b8f-ea6aece3be2b","Type":"ContainerDied","Data":"d87216af1786d31f63f788a321c7667ed828b154f234b14b91d08b1405a22a2c"}
Sep 11 22:15:04.097837 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:04.097721    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d87216af1786d31f63f788a321c7667ed828b154f234b14b91d08b1405a22a2c"
Sep 11 22:15:04.097837 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:04.097690    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75"
Sep 11 22:15:04.099112 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:04.099090    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819415-wrdw9" event={"ID":"0b28c946-2d0d-4188-8858-cdaf94fa2353","Type":"ContainerDied","Data":"c8ee7c9f51b73d35817b1ab3111dbe34781e256b3fbb8359d59bf8f65d9d2f81"}
Sep 11 22:15:04.099112 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:04.099107    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819415-wrdw9"
Sep 11 22:15:04.099253 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:04.099136    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8ee7c9f51b73d35817b1ab3111dbe34781e256b3fbb8359d59bf8f65d9d2f81"
Sep 11 22:15:10.570834 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:10.570796    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:10.930476 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:10.930439    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:16.941522 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:16.941484    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:17.043807 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:17.043774    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:17.223883 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:17.223789    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:18.491914 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:18.491874    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:18.597142 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:18.597099    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:19.266159 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:19.266113    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:19.362862 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:19.362832    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:19.463057 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:19.463021    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:20.747569 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:20.747533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:22.184183 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:22.184147    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:26.500363 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:26.500321    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:26.599925 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:26.599888    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:27.218679 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:27.218643    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:33.592585 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:33.592544    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:33.658603 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:33.658559    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:33.751785 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:33.751752    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:33.853540 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:33.853466    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:41.403232 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:41.403196    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:41.463519 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:41.463472    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:44.292199 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.292157    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:44.515167 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.515131    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:44.684699 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.684665    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"]
Sep 11 22:15:44.685118 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.685105    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b28c946-2d0d-4188-8858-cdaf94fa2353" containerName="cleanup"
Sep 11 22:15:44.685168 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.685119    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b28c946-2d0d-4188-8858-cdaf94fa2353" containerName="cleanup"
Sep 11 22:15:44.685168 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.685128    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ae0db7-4d02-4488-be6c-b4ea556f9239" containerName="cleanup"
Sep 11 22:15:44.685168 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.685134    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ae0db7-4d02-4488-be6c-b4ea556f9239" containerName="cleanup"
Sep 11 22:15:44.685168 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.685160    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2110600-2cd7-440c-8b8f-ea6aece3be2b" containerName="cleanup"
Sep 11 22:15:44.685168 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.685165    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2110600-2cd7-440c-8b8f-ea6aece3be2b" containerName="cleanup"
Sep 11 22:15:44.685386 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.685229    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ae0db7-4d02-4488-be6c-b4ea556f9239" containerName="cleanup"
Sep 11 22:15:44.685386 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.685241    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2110600-2cd7-440c-8b8f-ea6aece3be2b" containerName="cleanup"
Sep 11 22:15:44.685386 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.685250    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b28c946-2d0d-4188-8858-cdaf94fa2353" containerName="cleanup"
Sep 11 22:15:44.688271 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.688255    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.691092 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.691072    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-95d6c2\"/\"embedding-w0-95d6c2-kserve-self-signed-certs\""
Sep 11 22:15:44.696110 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.696089    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"]
Sep 11 22:15:44.767446 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.767407    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:44.789665 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.789642    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d2f722-f008-40e1-81d2-d7514abd5ad7-tls-certs\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.789790 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.789723    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpcmr\" (UniqueName: \"kubernetes.io/projected/c0d2f722-f008-40e1-81d2-d7514abd5ad7-kube-api-access-hpcmr\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.789790 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.789751    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-dshm\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.789790 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.789771    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-tmp-dir\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.790032 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.789897    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-home\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.790032 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.789936    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-model-cache\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.891174 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.891143    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-model-cache\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.891338 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.891183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d2f722-f008-40e1-81d2-d7514abd5ad7-tls-certs\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.891338 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.891240    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpcmr\" (UniqueName: \"kubernetes.io/projected/c0d2f722-f008-40e1-81d2-d7514abd5ad7-kube-api-access-hpcmr\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.891338 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.891266    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-dshm\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.891338 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.891282    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-tmp-dir\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.891511 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.891370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-home\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.891619 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.891595    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-model-cache\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.891726 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.891696    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-home\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.891877 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.891858    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-tmp-dir\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.893458 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.893441    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-dshm\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.893802 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.893784    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d2f722-f008-40e1-81d2-d7514abd5ad7-tls-certs\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.900411 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.900388    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpcmr\" (UniqueName: \"kubernetes.io/projected/c0d2f722-f008-40e1-81d2-d7514abd5ad7-kube-api-access-hpcmr\") pod \"embedding-w0-95d6c2-kserve-888b77975-6d8fr\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") " pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:44.997740 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:44.997683    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:45.279792 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:45.279766    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"]
Sep 11 22:15:45.281412 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:15:45.281374    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0d2f722_f008_40e1_81d2_d7514abd5ad7.slice/crio-f885d18499671c379c7ee3e0c63d23cda4ca0b9c2c038c4a30fe1eee389df6c7 WatchSource:0}: Error finding container f885d18499671c379c7ee3e0c63d23cda4ca0b9c2c038c4a30fe1eee389df6c7: Status 404 returned error can't find the container with id f885d18499671c379c7ee3e0c63d23cda4ca0b9c2c038c4a30fe1eee389df6c7
Sep 11 22:15:45.323138 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:45.323107    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr" event={"ID":"c0d2f722-f008-40e1-81d2-d7514abd5ad7","Type":"ContainerStarted","Data":"f885d18499671c379c7ee3e0c63d23cda4ca0b9c2c038c4a30fe1eee389df6c7"}
Sep 11 22:15:46.330322 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:46.330283    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr" event={"ID":"c0d2f722-f008-40e1-81d2-d7514abd5ad7","Type":"ContainerStarted","Data":"8684c73bbd5a5b5c8884722a297bfa51bd017f49d4ea8e2c858b5bf3f5b4e143"}
Sep 11 22:15:46.349726 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:46.349677    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr" podStartSLOduration=2.349662072 podStartE2EDuration="2.349662072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:15:46.347582285 +0000 UTC m=+1413.082693632" watchObservedRunningTime="2026-09-11 22:15:46.349662072 +0000 UTC m=+1413.084773418"
Sep 11 22:15:49.259589 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:49.259545    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:54.257039 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:54.257003    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:15:54.997865 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:54.997833    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:54.998063 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:54.997881    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:55.011482 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:55.011449    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:15:55.388487 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:15:55.388460    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:16:05.447026 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:05.446993    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:11.520099 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:11.519991    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:11.585736 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:11.585702    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:11.888808 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:11.888768    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:12.176108 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:12.176011    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:12.708511 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:12.708475    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:14.430990 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:14.430949    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:15.212163 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:15.212131    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:15.309383 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:15.309354    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:16.017244 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:16.017209    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:16.935788 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:16.935752    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:17.987794 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:17.987760    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:18.907952 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:18.907917    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:19.602637 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:19.602609    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:20.404857 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:20.404821    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:20.505862 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:20.505826    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:20.611280 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:20.611244    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:20.875808 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:20.875760    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:20.976729 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:20.976693    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:20.993668 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:20.993638    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"]
Sep 11 22:16:20.997861 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:20.997845    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"
Sep 11 22:16:21.000789 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:21.000766    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fe6196\""
Sep 11 22:16:21.000880 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:21.000791    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fe6196-dockercfg-8tkfc\""
Sep 11 22:16:21.014192 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:21.014168    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"]
Sep 11 22:16:21.133207 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:21.133126    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9pqg\" (UniqueName: \"kubernetes.io/projected/624908c2-bcdf-4891-b43a-9e1c33247e62-kube-api-access-m9pqg\") pod \"payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8\" (UID: \"624908c2-bcdf-4891-b43a-9e1c33247e62\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"
Sep 11 22:16:21.133389 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:21.133336    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624908c2-bcdf-4891-b43a-9e1c33247e62-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8\" (UID: \"624908c2-bcdf-4891-b43a-9e1c33247e62\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"
Sep 11 22:16:21.234511 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:21.234471    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624908c2-bcdf-4891-b43a-9e1c33247e62-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8\" (UID: \"624908c2-bcdf-4891-b43a-9e1c33247e62\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"
Sep 11 22:16:21.234688 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:21.234531    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9pqg\" (UniqueName: \"kubernetes.io/projected/624908c2-bcdf-4891-b43a-9e1c33247e62-kube-api-access-m9pqg\") pod \"payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8\" (UID: \"624908c2-bcdf-4891-b43a-9e1c33247e62\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"
Sep 11 22:16:21.235167 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:21.235146    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624908c2-bcdf-4891-b43a-9e1c33247e62-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8\" (UID: \"624908c2-bcdf-4891-b43a-9e1c33247e62\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"
Sep 11 22:16:21.242527 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:21.242501    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9pqg\" (UniqueName: \"kubernetes.io/projected/624908c2-bcdf-4891-b43a-9e1c33247e62-kube-api-access-m9pqg\") pod \"payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8\" (UID: \"624908c2-bcdf-4891-b43a-9e1c33247e62\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"
Sep 11 22:16:21.308021 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:21.307993    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"
Sep 11 22:16:21.580819 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:21.580790    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"]
Sep 11 22:16:21.582285 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:16:21.582262    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624908c2_bcdf_4891_b43a_9e1c33247e62.slice/crio-62a0488a4fd991e26ea50559173e64c712711153986fe95967979b36ea0162c4 WatchSource:0}: Error finding container 62a0488a4fd991e26ea50559173e64c712711153986fe95967979b36ea0162c4: Status 404 returned error can't find the container with id 62a0488a4fd991e26ea50559173e64c712711153986fe95967979b36ea0162c4
Sep 11 22:16:22.524451 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:22.524419    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8" event={"ID":"624908c2-bcdf-4891-b43a-9e1c33247e62","Type":"ContainerStarted","Data":"bbe32e54ef0f5897893f5f46112fda72e7340652f060b5f018f325fb94339ac6"}
Sep 11 22:16:22.524858 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:22.524471    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8" event={"ID":"624908c2-bcdf-4891-b43a-9e1c33247e62","Type":"ContainerStarted","Data":"62a0488a4fd991e26ea50559173e64c712711153986fe95967979b36ea0162c4"}
Sep 11 22:16:22.524858 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:22.524509    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"
Sep 11 22:16:22.543951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:22.543896    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8" podStartSLOduration=2.543876482 podStartE2EDuration="2.543876482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:16:22.542007358 +0000 UTC m=+1449.277118705" watchObservedRunningTime="2026-09-11 22:16:22.543876482 +0000 UTC m=+1449.278987830"
Sep 11 22:16:33.531755 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:33.531725    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"
Sep 11 22:16:41.002936 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:41.002896    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:41.811801 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:41.811757    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:43.240209 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:43.240170    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:43.958532 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:43.958491    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:44.715720 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:44.715676    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:45.190071 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:45.190030    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:45.408153 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:45.408101    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:45.648872 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:45.648825    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:45.976070 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:45.975973    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:46.315727 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:46.315638    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:46.539506 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:46.539478    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:47.838659 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:47.838619    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:47.940354 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:47.940266    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:48.890371 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:48.890332    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:49.318350 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:49.318243    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:49.416685 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:49.416648    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:49.778890 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:49.778853    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:50.826363 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:50.826329    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:51.723691 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:51.723657    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:52.434271 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:52.434240    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:53.142276 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:53.142234    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:54.097198 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:54.097158    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:54.404489 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:54.404396    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:55.006799 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.006762    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:55.682857 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.682822    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"]
Sep 11 22:16:55.686995 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.686971    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:55.689771 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.689744    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2715d7-serving-cert\""
Sep 11 22:16:55.696103 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.696079    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"]
Sep 11 22:16:55.716423 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.716396    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:55.777256 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.777225    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:55.864284 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.864250    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrst4\" (UniqueName: \"kubernetes.io/projected/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-kube-api-access-lrst4\") pod \"maas-api-e2e-api-b-2715d7-579cc74894-zcjxp\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:55.864482 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.864301    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-maas-api-tls\") pod \"maas-api-e2e-api-b-2715d7-579cc74894-zcjxp\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:55.864482 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.864361    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-metrics-certs\") pod \"maas-api-e2e-api-b-2715d7-579cc74894-zcjxp\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:55.864612 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.864482    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-2715d7-579cc74894-zcjxp\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:55.877587 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.877556    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:55.965457 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.965382    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-maas-api-tls\") pod \"maas-api-e2e-api-b-2715d7-579cc74894-zcjxp\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:55.965457 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.965418    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-metrics-certs\") pod \"maas-api-e2e-api-b-2715d7-579cc74894-zcjxp\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:55.965654 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.965472    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-2715d7-579cc74894-zcjxp\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:55.965654 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.965541    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrst4\" (UniqueName: \"kubernetes.io/projected/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-kube-api-access-lrst4\") pod \"maas-api-e2e-api-b-2715d7-579cc74894-zcjxp\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:55.966202 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.966177    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-2715d7-579cc74894-zcjxp\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:55.968033 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.968003    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-maas-api-tls\") pod \"maas-api-e2e-api-b-2715d7-579cc74894-zcjxp\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:55.968033 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.968018    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-metrics-certs\") pod \"maas-api-e2e-api-b-2715d7-579cc74894-zcjxp\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:55.973886 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.973869    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrst4\" (UniqueName: \"kubernetes.io/projected/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-kube-api-access-lrst4\") pod \"maas-api-e2e-api-b-2715d7-579cc74894-zcjxp\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:55.999956 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:55.999937    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:56.286966 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.286882    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:56.294577 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.294554    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"]
Sep 11 22:16:56.297467 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:16:56.297436    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ab6483_77d4_4811_aa2a_7d9ee59f18a7.slice/crio-916ce2bf1797f8087f1bc422d9bbe57b7faf3490aecc3c993614fddeb36a1745 WatchSource:0}: Error finding container 916ce2bf1797f8087f1bc422d9bbe57b7faf3490aecc3c993614fddeb36a1745: Status 404 returned error can't find the container with id 916ce2bf1797f8087f1bc422d9bbe57b7faf3490aecc3c993614fddeb36a1745
Sep 11 22:16:56.298915 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.298897    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:16:56.379622 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.379591    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:56.426220 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.424837    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"]
Sep 11 22:16:56.430142 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.430122    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"
Sep 11 22:16:56.433473 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.433451    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2715d7-dockercfg-hr44v\""
Sep 11 22:16:56.433640 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.433622    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2715d7\""
Sep 11 22:16:56.438718 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.438695    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"]
Sep 11 22:16:56.550544 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.550459    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"]
Sep 11 22:16:56.554517 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.554492    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"
Sep 11 22:16:56.565765 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.565723    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"]
Sep 11 22:16:56.571589 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.571537    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl5hk\" (UniqueName: \"kubernetes.io/projected/1980f758-2442-4e7d-91f5-52d2a262630a-kube-api-access-wl5hk\") pod \"payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq\" (UID: \"1980f758-2442-4e7d-91f5-52d2a262630a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"
Sep 11 22:16:56.571727 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.571650    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1980f758-2442-4e7d-91f5-52d2a262630a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq\" (UID: \"1980f758-2442-4e7d-91f5-52d2a262630a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"
Sep 11 22:16:56.673038 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.673011    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl5hk\" (UniqueName: \"kubernetes.io/projected/1980f758-2442-4e7d-91f5-52d2a262630a-kube-api-access-wl5hk\") pod \"payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq\" (UID: \"1980f758-2442-4e7d-91f5-52d2a262630a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"
Sep 11 22:16:56.673171 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.673055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beeb9ee-ace8-41d1-98e1-d227a168f7b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn\" (UID: \"4beeb9ee-ace8-41d1-98e1-d227a168f7b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"
Sep 11 22:16:56.673171 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.673114    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1980f758-2442-4e7d-91f5-52d2a262630a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq\" (UID: \"1980f758-2442-4e7d-91f5-52d2a262630a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"
Sep 11 22:16:56.673291 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.673207    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpr46\" (UniqueName: \"kubernetes.io/projected/4beeb9ee-ace8-41d1-98e1-d227a168f7b5-kube-api-access-tpr46\") pod \"payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn\" (UID: \"4beeb9ee-ace8-41d1-98e1-d227a168f7b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"
Sep 11 22:16:56.673862 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.673832    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1980f758-2442-4e7d-91f5-52d2a262630a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq\" (UID: \"1980f758-2442-4e7d-91f5-52d2a262630a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"
Sep 11 22:16:56.681049 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.681026    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl5hk\" (UniqueName: \"kubernetes.io/projected/1980f758-2442-4e7d-91f5-52d2a262630a-kube-api-access-wl5hk\") pod \"payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq\" (UID: \"1980f758-2442-4e7d-91f5-52d2a262630a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"
Sep 11 22:16:56.713968 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.713943    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp" event={"ID":"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7","Type":"ContainerStarted","Data":"916ce2bf1797f8087f1bc422d9bbe57b7faf3490aecc3c993614fddeb36a1745"}
Sep 11 22:16:56.730323 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.730281    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:56.742587 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.742563    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"
Sep 11 22:16:56.774690 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.774657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beeb9ee-ace8-41d1-98e1-d227a168f7b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn\" (UID: \"4beeb9ee-ace8-41d1-98e1-d227a168f7b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"
Sep 11 22:16:56.774831 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.774766    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpr46\" (UniqueName: \"kubernetes.io/projected/4beeb9ee-ace8-41d1-98e1-d227a168f7b5-kube-api-access-tpr46\") pod \"payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn\" (UID: \"4beeb9ee-ace8-41d1-98e1-d227a168f7b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"
Sep 11 22:16:56.775340 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.775322    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beeb9ee-ace8-41d1-98e1-d227a168f7b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn\" (UID: \"4beeb9ee-ace8-41d1-98e1-d227a168f7b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"
Sep 11 22:16:56.783236 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.783214    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpr46\" (UniqueName: \"kubernetes.io/projected/4beeb9ee-ace8-41d1-98e1-d227a168f7b5-kube-api-access-tpr46\") pod \"payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn\" (UID: \"4beeb9ee-ace8-41d1-98e1-d227a168f7b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"
Sep 11 22:16:56.868116 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:56.868090    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"
Sep 11 22:16:57.017150 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.017116    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"]
Sep 11 22:16:57.018002 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:16:57.017975    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1980f758_2442_4e7d_91f5_52d2a262630a.slice/crio-665e96b7d7931dbe61a0123a29c40ff548a944edfdd2888fb60e4838c9c6d437 WatchSource:0}: Error finding container 665e96b7d7931dbe61a0123a29c40ff548a944edfdd2888fb60e4838c9c6d437: Status 404 returned error can't find the container with id 665e96b7d7931dbe61a0123a29c40ff548a944edfdd2888fb60e4838c9c6d437
Sep 11 22:16:57.145054 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.145025    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"]
Sep 11 22:16:57.146423 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:16:57.146386    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4beeb9ee_ace8_41d1_98e1_d227a168f7b5.slice/crio-d7905a5590b94f23a1f8c2144ec99fdea0af246439ede9e055ce15f388d82dcb WatchSource:0}: Error finding container d7905a5590b94f23a1f8c2144ec99fdea0af246439ede9e055ce15f388d82dcb: Status 404 returned error can't find the container with id d7905a5590b94f23a1f8c2144ec99fdea0af246439ede9e055ce15f388d82dcb
Sep 11 22:16:57.509954 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.509872    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:57.723973 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.723944    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"
Sep 11 22:16:57.724404 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.724003    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq" event={"ID":"1980f758-2442-4e7d-91f5-52d2a262630a","Type":"ContainerStarted","Data":"3687ce4476cd43cd8ecc05107b2aaab9f79205219893b1425db48285277eef4a"}
Sep 11 22:16:57.724404 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.724030    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq" event={"ID":"1980f758-2442-4e7d-91f5-52d2a262630a","Type":"ContainerStarted","Data":"665e96b7d7931dbe61a0123a29c40ff548a944edfdd2888fb60e4838c9c6d437"}
Sep 11 22:16:57.724707 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.724676    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp" event={"ID":"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7","Type":"ContainerStarted","Data":"1f82e83bd0f9987ddeb52e381674a6bd59736690356b0ced8139dbd611ba1990"}
Sep 11 22:16:57.724802 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.724774    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:16:57.726122 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.726101    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn" event={"ID":"4beeb9ee-ace8-41d1-98e1-d227a168f7b5","Type":"ContainerStarted","Data":"6c9fe08f5a396b266df50f3df526733fc6bb887958ef9228762fd236bd378a54"}
Sep 11 22:16:57.726227 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.726140    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn" event={"ID":"4beeb9ee-ace8-41d1-98e1-d227a168f7b5","Type":"ContainerStarted","Data":"d7905a5590b94f23a1f8c2144ec99fdea0af246439ede9e055ce15f388d82dcb"}
Sep 11 22:16:57.726267 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.726235    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"
Sep 11 22:16:57.741389 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.741347    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq" podStartSLOduration=1.741305114 podStartE2EDuration="1.741305114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:16:57.740777177 +0000 UTC m=+1484.475888524" watchObservedRunningTime="2026-09-11 22:16:57.741305114 +0000 UTC m=+1484.476416465"
Sep 11 22:16:57.760658 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.760585    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp" podStartSLOduration=2.403406016 podStartE2EDuration="2.760569117s" totalImagesPullingTime="357.163101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:16:57.759064748 +0000 UTC m=+1484.494176096" watchObservedRunningTime="2026-09-11 22:16:57.760569117 +0000 UTC m=+1484.495680462"
Sep 11 22:16:57.776730 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:57.776696    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn" podStartSLOduration=1.7766838740000002 podStartE2EDuration="1.776683874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:16:57.776163734 +0000 UTC m=+1484.511275087" watchObservedRunningTime="2026-09-11 22:16:57.776683874 +0000 UTC m=+1484.511795216"
Sep 11 22:16:58.306233 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:58.306184    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:59.100105 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:59.100059    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:16:59.410734 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:59.410657    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"]
Sep 11 22:16:59.419570 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:59.419539    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"]
Sep 11 22:16:59.479829 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:16:59.479798    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:00.097521 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:00.097482    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:00.803027 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:00.802993    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:01.651582 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:01.651531    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:02.427592 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:02.427559    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:03.742579 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:03.742547    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:17:04.922724 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:04.922686    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:08.735510 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:08.735480    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"
Sep 11 22:17:09.740188 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:09.740158    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"
Sep 11 22:17:10.554564 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.554533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:10.561010 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.560986    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c"]
Sep 11 22:17:10.563220 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.563192    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-95d6c2-29819415-x928c"]
Sep 11 22:17:10.586885 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.586847    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"]
Sep 11 22:17:10.587112 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.587084    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp" podUID="b58d6418-db5f-41c3-9908-1547d5f7cdec" containerName="payload-processing" containerID="cri-o://1b1221bebe5bced4989fa05b8a9c161915c27a11647793c500acad80d68490df" gracePeriod=30
Sep 11 22:17:10.594547 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.594523    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"]
Sep 11 22:17:10.594749 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.594723    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s" podUID="1f436254-0576-490b-9d08-9968d66f3b85" containerName="payload-pre-processing" containerID="cri-o://5703eecc0fc57b914dcf5ad24a23ab77827176ff91cb2f3d9b74e4c0f1d102f8" gracePeriod=30
Sep 11 22:17:10.630023 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.629888    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:10.688146 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.688118    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:10.754705 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.754655    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:10.802606 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.802493    2574 generic.go:356] "Generic (PLEG): container finished" podID="1f436254-0576-490b-9d08-9968d66f3b85" containerID="5703eecc0fc57b914dcf5ad24a23ab77827176ff91cb2f3d9b74e4c0f1d102f8" exitCode=0
Sep 11 22:17:10.802606 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.802563    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s" event={"ID":"1f436254-0576-490b-9d08-9968d66f3b85","Type":"ContainerDied","Data":"5703eecc0fc57b914dcf5ad24a23ab77827176ff91cb2f3d9b74e4c0f1d102f8"}
Sep 11 22:17:10.805000 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.804719    2574 generic.go:356] "Generic (PLEG): container finished" podID="b58d6418-db5f-41c3-9908-1547d5f7cdec" containerID="1b1221bebe5bced4989fa05b8a9c161915c27a11647793c500acad80d68490df" exitCode=0
Sep 11 22:17:10.805000 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.804911    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp" event={"ID":"b58d6418-db5f-41c3-9908-1547d5f7cdec","Type":"ContainerDied","Data":"1b1221bebe5bced4989fa05b8a9c161915c27a11647793c500acad80d68490df"}
Sep 11 22:17:10.867639 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.867615    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"
Sep 11 22:17:10.902731 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.902707    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"
Sep 11 22:17:10.926777 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.926750    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bdwd\" (UniqueName: \"kubernetes.io/projected/1f436254-0576-490b-9d08-9968d66f3b85-kube-api-access-4bdwd\") pod \"1f436254-0576-490b-9d08-9968d66f3b85\" (UID: \"1f436254-0576-490b-9d08-9968d66f3b85\") "
Sep 11 22:17:10.926915 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.926805    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58d6418-db5f-41c3-9908-1547d5f7cdec-plugins-config-volume\") pod \"b58d6418-db5f-41c3-9908-1547d5f7cdec\" (UID: \"b58d6418-db5f-41c3-9908-1547d5f7cdec\") "
Sep 11 22:17:10.926915 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.926853    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nbdm\" (UniqueName: \"kubernetes.io/projected/b58d6418-db5f-41c3-9908-1547d5f7cdec-kube-api-access-4nbdm\") pod \"b58d6418-db5f-41c3-9908-1547d5f7cdec\" (UID: \"b58d6418-db5f-41c3-9908-1547d5f7cdec\") "
Sep 11 22:17:10.927020 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.926989    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f436254-0576-490b-9d08-9968d66f3b85-plugins-config-volume\") pod \"1f436254-0576-490b-9d08-9968d66f3b85\" (UID: \"1f436254-0576-490b-9d08-9968d66f3b85\") "
Sep 11 22:17:10.927391 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.927214    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b58d6418-db5f-41c3-9908-1547d5f7cdec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b58d6418-db5f-41c3-9908-1547d5f7cdec" (UID: "b58d6418-db5f-41c3-9908-1547d5f7cdec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:17:10.927391 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.927376    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58d6418-db5f-41c3-9908-1547d5f7cdec-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:10.927565 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.927447    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f436254-0576-490b-9d08-9968d66f3b85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1f436254-0576-490b-9d08-9968d66f3b85" (UID: "1f436254-0576-490b-9d08-9968d66f3b85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:17:10.929029 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.929005    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f436254-0576-490b-9d08-9968d66f3b85-kube-api-access-4bdwd" (OuterVolumeSpecName: "kube-api-access-4bdwd") pod "1f436254-0576-490b-9d08-9968d66f3b85" (UID: "1f436254-0576-490b-9d08-9968d66f3b85"). InnerVolumeSpecName "kube-api-access-4bdwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:10.929172 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:10.929159    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58d6418-db5f-41c3-9908-1547d5f7cdec-kube-api-access-4nbdm" (OuterVolumeSpecName: "kube-api-access-4nbdm") pod "b58d6418-db5f-41c3-9908-1547d5f7cdec" (UID: "b58d6418-db5f-41c3-9908-1547d5f7cdec"). InnerVolumeSpecName "kube-api-access-4nbdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:11.028137 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.028101    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f436254-0576-490b-9d08-9968d66f3b85-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:11.028137 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.028130    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bdwd\" (UniqueName: \"kubernetes.io/projected/1f436254-0576-490b-9d08-9968d66f3b85-kube-api-access-4bdwd\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:11.028137 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.028144    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nbdm\" (UniqueName: \"kubernetes.io/projected/b58d6418-db5f-41c3-9908-1547d5f7cdec-kube-api-access-4nbdm\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:11.095522 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.095492    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:11.201162 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.201127    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:11.721438 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.721406    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5ae0db7-4d02-4488-be6c-b4ea556f9239" path="/var/lib/kubelet/pods/a5ae0db7-4d02-4488-be6c-b4ea556f9239/volumes"
Sep 11 22:17:11.812122 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.812098    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"
Sep 11 22:17:11.812515 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.812141    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s" event={"ID":"1f436254-0576-490b-9d08-9968d66f3b85","Type":"ContainerDied","Data":"f98158c2037fbcfc1f3db587f02f164a349d2ba5a2b4742692edd2830aa46a53"}
Sep 11 22:17:11.812515 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.812204    2574 scope.go:117] "RemoveContainer" containerID="5703eecc0fc57b914dcf5ad24a23ab77827176ff91cb2f3d9b74e4c0f1d102f8"
Sep 11 22:17:11.814002 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.813832    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp" event={"ID":"b58d6418-db5f-41c3-9908-1547d5f7cdec","Type":"ContainerDied","Data":"34ea38a6591558cf09c9f0f59e484dc0329153d4bc913c839b1ff0e5566c7f39"}
Sep 11 22:17:11.814002 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.813917    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"
Sep 11 22:17:11.822991 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.822966    2574 scope.go:117] "RemoveContainer" containerID="1b1221bebe5bced4989fa05b8a9c161915c27a11647793c500acad80d68490df"
Sep 11 22:17:11.836092 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.836073    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"]
Sep 11 22:17:11.839663 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.839644    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-95d6c2-76b579d66-hwq8s"]
Sep 11 22:17:11.851925 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.851903    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"]
Sep 11 22:17:11.856145 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.856124    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-95d6c2-59669fbcbf-trvpp"]
Sep 11 22:17:11.915553 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:11.915492    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:12.685452 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:12.685420    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:13.572438 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:13.572402    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:13.722222 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:13.722193    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f436254-0576-490b-9d08-9968d66f3b85" path="/var/lib/kubelet/pods/1f436254-0576-490b-9d08-9968d66f3b85/volumes"
Sep 11 22:17:13.722608 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:13.722594    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b58d6418-db5f-41c3-9908-1547d5f7cdec" path="/var/lib/kubelet/pods/b58d6418-db5f-41c3-9908-1547d5f7cdec/volumes"
Sep 11 22:17:13.780591 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:13.780560    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:17:13.783211 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:13.783191    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:17:13.784865 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:13.784848    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:17:13.787265 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:13.787247    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:17:14.245791 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.245676    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:14.431054 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.431019    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr" podUID="c0d2f722-f008-40e1-81d2-d7514abd5ad7" containerName="main" containerID="cri-o://8684c73bbd5a5b5c8884722a297bfa51bd017f49d4ea8e2c858b5bf3f5b4e143" gracePeriod=45
Sep 11 22:17:14.440698 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.440672    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95" podUID="d291c1a9-5ba8-4c01-8509-9e2c1c911134" containerName="main" containerID="cri-o://27288e3246092a101f43c436665c9d890e800220d62f6f674481d08bfca22154" gracePeriod=45
Sep 11 22:17:14.693650 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.693627    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:17:14.721626 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.721601    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:17:14.765580 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.765505    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-home\") pod \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") "
Sep 11 22:17:14.765580 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.765556    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-home\") pod \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") "
Sep 11 22:17:14.765800 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.765593    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-model-cache\") pod \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") "
Sep 11 22:17:14.765800 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.765621    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-dshm\") pod \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") "
Sep 11 22:17:14.765800 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.765706    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-dshm\") pod \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") "
Sep 11 22:17:14.765800 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.765788    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-home" (OuterVolumeSpecName: "home") pod "d291c1a9-5ba8-4c01-8509-9e2c1c911134" (UID: "d291c1a9-5ba8-4c01-8509-9e2c1c911134"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.766728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.765883    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-home" (OuterVolumeSpecName: "home") pod "c0d2f722-f008-40e1-81d2-d7514abd5ad7" (UID: "c0d2f722-f008-40e1-81d2-d7514abd5ad7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.766728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.765996    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlmkh\" (UniqueName: \"kubernetes.io/projected/d291c1a9-5ba8-4c01-8509-9e2c1c911134-kube-api-access-xlmkh\") pod \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") "
Sep 11 22:17:14.766728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.766069    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d2f722-f008-40e1-81d2-d7514abd5ad7-tls-certs\") pod \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") "
Sep 11 22:17:14.766728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.766125    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-model-cache\") pod \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") "
Sep 11 22:17:14.766728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.766155    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-tmp-dir\") pod \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") "
Sep 11 22:17:14.766728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.766188    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-model-cache" (OuterVolumeSpecName: "model-cache") pod "d291c1a9-5ba8-4c01-8509-9e2c1c911134" (UID: "d291c1a9-5ba8-4c01-8509-9e2c1c911134"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.766728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.766217    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpcmr\" (UniqueName: \"kubernetes.io/projected/c0d2f722-f008-40e1-81d2-d7514abd5ad7-kube-api-access-hpcmr\") pod \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") "
Sep 11 22:17:14.766728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.766260    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d291c1a9-5ba8-4c01-8509-9e2c1c911134-tls-certs\") pod \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\" (UID: \"d291c1a9-5ba8-4c01-8509-9e2c1c911134\") "
Sep 11 22:17:14.766728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.766338    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-tmp-dir\") pod \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\" (UID: \"c0d2f722-f008-40e1-81d2-d7514abd5ad7\") "
Sep 11 22:17:14.766728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.766440    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0d2f722-f008-40e1-81d2-d7514abd5ad7" (UID: "c0d2f722-f008-40e1-81d2-d7514abd5ad7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.767388 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.766812    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d291c1a9-5ba8-4c01-8509-9e2c1c911134" (UID: "d291c1a9-5ba8-4c01-8509-9e2c1c911134"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.767388 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.767087    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0d2f722-f008-40e1-81d2-d7514abd5ad7" (UID: "c0d2f722-f008-40e1-81d2-d7514abd5ad7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.768465 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.768443    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-home\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.768569 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.768472    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-home\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.768569 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.768485    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-model-cache\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.768569 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.768499    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-model-cache\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.768569 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.768513    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-tmp-dir\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.768569 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.768524    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-tmp-dir\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.769322 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.769273    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d2f722-f008-40e1-81d2-d7514abd5ad7-kube-api-access-hpcmr" (OuterVolumeSpecName: "kube-api-access-hpcmr") pod "c0d2f722-f008-40e1-81d2-d7514abd5ad7" (UID: "c0d2f722-f008-40e1-81d2-d7514abd5ad7"). InnerVolumeSpecName "kube-api-access-hpcmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:14.769544 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.769450    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-dshm" (OuterVolumeSpecName: "dshm") pod "d291c1a9-5ba8-4c01-8509-9e2c1c911134" (UID: "d291c1a9-5ba8-4c01-8509-9e2c1c911134"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.769544 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.769505    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d291c1a9-5ba8-4c01-8509-9e2c1c911134-kube-api-access-xlmkh" (OuterVolumeSpecName: "kube-api-access-xlmkh") pod "d291c1a9-5ba8-4c01-8509-9e2c1c911134" (UID: "d291c1a9-5ba8-4c01-8509-9e2c1c911134"). InnerVolumeSpecName "kube-api-access-xlmkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:14.770300 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.770271    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-dshm" (OuterVolumeSpecName: "dshm") pod "c0d2f722-f008-40e1-81d2-d7514abd5ad7" (UID: "c0d2f722-f008-40e1-81d2-d7514abd5ad7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.770679 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.770659    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d2f722-f008-40e1-81d2-d7514abd5ad7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0d2f722-f008-40e1-81d2-d7514abd5ad7" (UID: "c0d2f722-f008-40e1-81d2-d7514abd5ad7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:17:14.770941 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.770911    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d291c1a9-5ba8-4c01-8509-9e2c1c911134-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d291c1a9-5ba8-4c01-8509-9e2c1c911134" (UID: "d291c1a9-5ba8-4c01-8509-9e2c1c911134"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:17:14.834435 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.834403    2574 generic.go:356] "Generic (PLEG): container finished" podID="d291c1a9-5ba8-4c01-8509-9e2c1c911134" containerID="27288e3246092a101f43c436665c9d890e800220d62f6f674481d08bfca22154" exitCode=0
Sep 11 22:17:14.834589 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.834447    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"
Sep 11 22:17:14.834589 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.834481    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95" event={"ID":"d291c1a9-5ba8-4c01-8509-9e2c1c911134","Type":"ContainerDied","Data":"27288e3246092a101f43c436665c9d890e800220d62f6f674481d08bfca22154"}
Sep 11 22:17:14.834589 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.834519    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95" event={"ID":"d291c1a9-5ba8-4c01-8509-9e2c1c911134","Type":"ContainerDied","Data":"f06a112e783c47ad7057f57b10c2e4671bdde1d7f6f9b2dc6f9c3fc6165c1d82"}
Sep 11 22:17:14.834589 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.834538    2574 scope.go:117] "RemoveContainer" containerID="27288e3246092a101f43c436665c9d890e800220d62f6f674481d08bfca22154"
Sep 11 22:17:14.836013 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.835988    2574 generic.go:356] "Generic (PLEG): container finished" podID="c0d2f722-f008-40e1-81d2-d7514abd5ad7" containerID="8684c73bbd5a5b5c8884722a297bfa51bd017f49d4ea8e2c858b5bf3f5b4e143" exitCode=0
Sep 11 22:17:14.836124 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.836022    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"
Sep 11 22:17:14.836124 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.836043    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr" event={"ID":"c0d2f722-f008-40e1-81d2-d7514abd5ad7","Type":"ContainerDied","Data":"8684c73bbd5a5b5c8884722a297bfa51bd017f49d4ea8e2c858b5bf3f5b4e143"}
Sep 11 22:17:14.836124 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.836075    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr" event={"ID":"c0d2f722-f008-40e1-81d2-d7514abd5ad7","Type":"ContainerDied","Data":"f885d18499671c379c7ee3e0c63d23cda4ca0b9c2c038c4a30fe1eee389df6c7"}
Sep 11 22:17:14.844240 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.844225    2574 scope.go:117] "RemoveContainer" containerID="27288e3246092a101f43c436665c9d890e800220d62f6f674481d08bfca22154"
Sep 11 22:17:14.844524 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:17:14.844502    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27288e3246092a101f43c436665c9d890e800220d62f6f674481d08bfca22154\": container with ID starting with 27288e3246092a101f43c436665c9d890e800220d62f6f674481d08bfca22154 not found: ID does not exist" containerID="27288e3246092a101f43c436665c9d890e800220d62f6f674481d08bfca22154"
Sep 11 22:17:14.844610 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.844532    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27288e3246092a101f43c436665c9d890e800220d62f6f674481d08bfca22154"} err="failed to get container status \"27288e3246092a101f43c436665c9d890e800220d62f6f674481d08bfca22154\": rpc error: code = NotFound desc = could not find container \"27288e3246092a101f43c436665c9d890e800220d62f6f674481d08bfca22154\": container with ID starting with 27288e3246092a101f43c436665c9d890e800220d62f6f674481d08bfca22154 not found: ID does not exist"
Sep 11 22:17:14.844610 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.844554    2574 scope.go:117] "RemoveContainer" containerID="8684c73bbd5a5b5c8884722a297bfa51bd017f49d4ea8e2c858b5bf3f5b4e143"
Sep 11 22:17:14.852585 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.852568    2574 scope.go:117] "RemoveContainer" containerID="8684c73bbd5a5b5c8884722a297bfa51bd017f49d4ea8e2c858b5bf3f5b4e143"
Sep 11 22:17:14.852839 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:17:14.852822    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8684c73bbd5a5b5c8884722a297bfa51bd017f49d4ea8e2c858b5bf3f5b4e143\": container with ID starting with 8684c73bbd5a5b5c8884722a297bfa51bd017f49d4ea8e2c858b5bf3f5b4e143 not found: ID does not exist" containerID="8684c73bbd5a5b5c8884722a297bfa51bd017f49d4ea8e2c858b5bf3f5b4e143"
Sep 11 22:17:14.852889 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.852844    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8684c73bbd5a5b5c8884722a297bfa51bd017f49d4ea8e2c858b5bf3f5b4e143"} err="failed to get container status \"8684c73bbd5a5b5c8884722a297bfa51bd017f49d4ea8e2c858b5bf3f5b4e143\": rpc error: code = NotFound desc = could not find container \"8684c73bbd5a5b5c8884722a297bfa51bd017f49d4ea8e2c858b5bf3f5b4e143\": container with ID starting with 8684c73bbd5a5b5c8884722a297bfa51bd017f49d4ea8e2c858b5bf3f5b4e143 not found: ID does not exist"
Sep 11 22:17:14.857784 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.857762    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"]
Sep 11 22:17:14.861959 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.861939    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/premium-w0-95d6c2-kserve-6995c8b755-dqk95"]
Sep 11 22:17:14.869456 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.869436    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d2f722-f008-40e1-81d2-d7514abd5ad7-dshm\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.869551 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.869459    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d291c1a9-5ba8-4c01-8509-9e2c1c911134-dshm\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.869551 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.869472    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlmkh\" (UniqueName: \"kubernetes.io/projected/d291c1a9-5ba8-4c01-8509-9e2c1c911134-kube-api-access-xlmkh\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.869551 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.869486    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d2f722-f008-40e1-81d2-d7514abd5ad7-tls-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.869551 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.869500    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpcmr\" (UniqueName: \"kubernetes.io/projected/c0d2f722-f008-40e1-81d2-d7514abd5ad7-kube-api-access-hpcmr\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.869551 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.869513    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d291c1a9-5ba8-4c01-8509-9e2c1c911134-tls-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.873541 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.873521    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"]
Sep 11 22:17:14.885594 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.885567    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/embedding-w0-95d6c2-kserve-888b77975-6d8fr"]
Sep 11 22:17:14.927154 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:14.927129    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:15.703137 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:15.703047    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:15.723356 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:15.723302    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0d2f722-f008-40e1-81d2-d7514abd5ad7" path="/var/lib/kubelet/pods/c0d2f722-f008-40e1-81d2-d7514abd5ad7/volumes"
Sep 11 22:17:15.723762 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:15.723747    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d291c1a9-5ba8-4c01-8509-9e2c1c911134" path="/var/lib/kubelet/pods/d291c1a9-5ba8-4c01-8509-9e2c1c911134/volumes"
Sep 11 22:17:16.403176 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:16.403136    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:17.180084 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:17.180045    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:17.444553 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:17.444464    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:17.552415 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:17.552377    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:17.763761 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:17.763645    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:18.154607 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:18.154572    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:19.183680 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:19.183644    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:19.404823 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:19.404765    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:20.026016 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:20.025980    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:20.831386 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:20.831352    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:21.708652 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:21.708614    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:22.613347 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:22.613296    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:23.532917 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:23.532856    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:24.375504 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:24.375467    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:25.031429 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:25.031388    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:25.697253 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:25.697221    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:26.500980 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:26.500931    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:27.159756 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:27.159719    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:27.904320 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:27.904270    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:28.725809 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:28.725767    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:32.533437 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:32.533399    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:36.527596 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:36.527507    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:36.628096 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:36.628057    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:36.988872 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:36.988838    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:38.198754 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.198714    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:38.232556 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.232525    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"]
Sep 11 22:17:38.232991 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.232958    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8" podUID="624908c2-bcdf-4891-b43a-9e1c33247e62" containerName="payload-processing" containerID="cri-o://bbe32e54ef0f5897893f5f46112fda72e7340652f060b5f018f325fb94339ac6" gracePeriod=30
Sep 11 22:17:38.299559 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.299528    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:38.401442 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.401412    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:38.469466 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.469335    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:38.496699 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.496674    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"
Sep 11 22:17:38.501195 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.501167    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624908c2-bcdf-4891-b43a-9e1c33247e62-plugins-config-volume\") pod \"624908c2-bcdf-4891-b43a-9e1c33247e62\" (UID: \"624908c2-bcdf-4891-b43a-9e1c33247e62\") "
Sep 11 22:17:38.501349 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.501229    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9pqg\" (UniqueName: \"kubernetes.io/projected/624908c2-bcdf-4891-b43a-9e1c33247e62-kube-api-access-m9pqg\") pod \"624908c2-bcdf-4891-b43a-9e1c33247e62\" (UID: \"624908c2-bcdf-4891-b43a-9e1c33247e62\") "
Sep 11 22:17:38.501613 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.501584    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624908c2-bcdf-4891-b43a-9e1c33247e62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "624908c2-bcdf-4891-b43a-9e1c33247e62" (UID: "624908c2-bcdf-4891-b43a-9e1c33247e62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:17:38.503523 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.503503    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624908c2-bcdf-4891-b43a-9e1c33247e62-kube-api-access-m9pqg" (OuterVolumeSpecName: "kube-api-access-m9pqg") pod "624908c2-bcdf-4891-b43a-9e1c33247e62" (UID: "624908c2-bcdf-4891-b43a-9e1c33247e62"). InnerVolumeSpecName "kube-api-access-m9pqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:38.602979 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.602941    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624908c2-bcdf-4891-b43a-9e1c33247e62-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:38.602979 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.602973    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9pqg\" (UniqueName: \"kubernetes.io/projected/624908c2-bcdf-4891-b43a-9e1c33247e62-kube-api-access-m9pqg\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:17:38.761334 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.761213    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:38.909221 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.909184    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:38.976914 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.976884    2574 generic.go:356] "Generic (PLEG): container finished" podID="624908c2-bcdf-4891-b43a-9e1c33247e62" containerID="bbe32e54ef0f5897893f5f46112fda72e7340652f060b5f018f325fb94339ac6" exitCode=0
Sep 11 22:17:38.977060 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.976926    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"
Sep 11 22:17:38.977060 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.976957    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8" event={"ID":"624908c2-bcdf-4891-b43a-9e1c33247e62","Type":"ContainerDied","Data":"bbe32e54ef0f5897893f5f46112fda72e7340652f060b5f018f325fb94339ac6"}
Sep 11 22:17:38.977060 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.977011    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8" event={"ID":"624908c2-bcdf-4891-b43a-9e1c33247e62","Type":"ContainerDied","Data":"62a0488a4fd991e26ea50559173e64c712711153986fe95967979b36ea0162c4"}
Sep 11 22:17:38.977060 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.977037    2574 scope.go:117] "RemoveContainer" containerID="bbe32e54ef0f5897893f5f46112fda72e7340652f060b5f018f325fb94339ac6"
Sep 11 22:17:38.986589 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.986571    2574 scope.go:117] "RemoveContainer" containerID="bbe32e54ef0f5897893f5f46112fda72e7340652f060b5f018f325fb94339ac6"
Sep 11 22:17:38.986835 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:17:38.986818    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe32e54ef0f5897893f5f46112fda72e7340652f060b5f018f325fb94339ac6\": container with ID starting with bbe32e54ef0f5897893f5f46112fda72e7340652f060b5f018f325fb94339ac6 not found: ID does not exist" containerID="bbe32e54ef0f5897893f5f46112fda72e7340652f060b5f018f325fb94339ac6"
Sep 11 22:17:38.986881 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:38.986843    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe32e54ef0f5897893f5f46112fda72e7340652f060b5f018f325fb94339ac6"} err="failed to get container status \"bbe32e54ef0f5897893f5f46112fda72e7340652f060b5f018f325fb94339ac6\": rpc error: code = NotFound desc = could not find container \"bbe32e54ef0f5897893f5f46112fda72e7340652f060b5f018f325fb94339ac6\": container with ID starting with bbe32e54ef0f5897893f5f46112fda72e7340652f060b5f018f325fb94339ac6 not found: ID does not exist"
Sep 11 22:17:39.006782 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:39.006755    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"]
Sep 11 22:17:39.008043 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:39.008018    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe6196-79ff746c7-l9mx8"]
Sep 11 22:17:39.711949 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:39.711895    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:39.720770 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:39.720743    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="624908c2-bcdf-4891-b43a-9e1c33247e62" path="/var/lib/kubelet/pods/624908c2-bcdf-4891-b43a-9e1c33247e62/volumes"
Sep 11 22:17:40.396374 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:40.396337    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:41.049422 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:41.049390    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:41.925062 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:41.925007    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:42.561932 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:42.561904    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:43.204400 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:43.204358    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:44.599286 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:44.599245    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:44.699531 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:44.699496    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:44.863023 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:44.862941    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:45.918578 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:45.918529    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:46.709195 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:46.709164    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:47.603374 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:47.603334    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:48.244657 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:48.244627    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:49.018822 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:49.018785    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:49.643043 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:49.643012    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:50.396935 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:50.396900    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:52.037562 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:52.037516    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:53.239557 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:53.239520    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:54.531682 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:54.531613    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:55.120648 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:55.120587    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:55.782425 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:55.782393    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:56.517852 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:56.517821    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:57.341056 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:57.341023    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:58.109419 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:58.109383    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:58.232939 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:58.232910    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"]
Sep 11 22:17:58.302150 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:58.302115    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:17:58.917821 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:17:58.917782    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:04.484178 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.483600    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"]
Sep 11 22:18:04.484178 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.483853    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp" podUID="e8ab6483-77d4-4811-aa2a-7d9ee59f18a7" containerName="maas-api" containerID="cri-o://1f82e83bd0f9987ddeb52e381674a6bd59736690356b0ced8139dbd611ba1990" gracePeriod=30
Sep 11 22:18:04.512099 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.512056    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:04.540468 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.540436    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"]
Sep 11 22:18:04.540734 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.540649    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn" podUID="4beeb9ee-ace8-41d1-98e1-d227a168f7b5" containerName="payload-processing" containerID="cri-o://6c9fe08f5a396b266df50f3df526733fc6bb887958ef9228762fd236bd378a54" gracePeriod=30
Sep 11 22:18:04.548573 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.548401    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"]
Sep 11 22:18:04.548707 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.548635    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq" podUID="1980f758-2442-4e7d-91f5-52d2a262630a" containerName="payload-pre-processing" containerID="cri-o://3687ce4476cd43cd8ecc05107b2aaab9f79205219893b1425db48285277eef4a" gracePeriod=30
Sep 11 22:18:04.585024 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.584990    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:04.645509 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.645461    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:04.708031 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.707996    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:04.768504 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.768472    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:04.818132 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.818106    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"
Sep 11 22:18:04.853131 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.853109    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:18:04.886611 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.886586    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"
Sep 11 22:18:04.953188 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.953164    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-metrics-certs\") pod \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") "
Sep 11 22:18:04.953367 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.953197    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpr46\" (UniqueName: \"kubernetes.io/projected/4beeb9ee-ace8-41d1-98e1-d227a168f7b5-kube-api-access-tpr46\") pod \"4beeb9ee-ace8-41d1-98e1-d227a168f7b5\" (UID: \"4beeb9ee-ace8-41d1-98e1-d227a168f7b5\") "
Sep 11 22:18:04.953367 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.953264    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrst4\" (UniqueName: \"kubernetes.io/projected/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-kube-api-access-lrst4\") pod \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") "
Sep 11 22:18:04.953367 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.953284    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1980f758-2442-4e7d-91f5-52d2a262630a-plugins-config-volume\") pod \"1980f758-2442-4e7d-91f5-52d2a262630a\" (UID: \"1980f758-2442-4e7d-91f5-52d2a262630a\") "
Sep 11 22:18:04.953367 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.953357    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-maas-api-tls\") pod \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") "
Sep 11 22:18:04.953578 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.953412    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beeb9ee-ace8-41d1-98e1-d227a168f7b5-plugins-config-volume\") pod \"4beeb9ee-ace8-41d1-98e1-d227a168f7b5\" (UID: \"4beeb9ee-ace8-41d1-98e1-d227a168f7b5\") "
Sep 11 22:18:04.953578 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.953527    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl5hk\" (UniqueName: \"kubernetes.io/projected/1980f758-2442-4e7d-91f5-52d2a262630a-kube-api-access-wl5hk\") pod \"1980f758-2442-4e7d-91f5-52d2a262630a\" (UID: \"1980f758-2442-4e7d-91f5-52d2a262630a\") "
Sep 11 22:18:04.953688 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.953605    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-openshift-service-ca\") pod \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\" (UID: \"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7\") "
Sep 11 22:18:04.953742 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.953701    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1980f758-2442-4e7d-91f5-52d2a262630a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1980f758-2442-4e7d-91f5-52d2a262630a" (UID: "1980f758-2442-4e7d-91f5-52d2a262630a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:04.953809 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.953781    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4beeb9ee-ace8-41d1-98e1-d227a168f7b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4beeb9ee-ace8-41d1-98e1-d227a168f7b5" (UID: "4beeb9ee-ace8-41d1-98e1-d227a168f7b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:04.954176 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.954155    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1980f758-2442-4e7d-91f5-52d2a262630a-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:04.954281 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.954186    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beeb9ee-ace8-41d1-98e1-d227a168f7b5-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:04.954281 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.954156    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e8ab6483-77d4-4811-aa2a-7d9ee59f18a7" (UID: "e8ab6483-77d4-4811-aa2a-7d9ee59f18a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:04.955612 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.955591    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4beeb9ee-ace8-41d1-98e1-d227a168f7b5-kube-api-access-tpr46" (OuterVolumeSpecName: "kube-api-access-tpr46") pod "4beeb9ee-ace8-41d1-98e1-d227a168f7b5" (UID: "4beeb9ee-ace8-41d1-98e1-d227a168f7b5"). InnerVolumeSpecName "kube-api-access-tpr46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:04.955898 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.955877    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e8ab6483-77d4-4811-aa2a-7d9ee59f18a7" (UID: "e8ab6483-77d4-4811-aa2a-7d9ee59f18a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:04.955965 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.955911    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e8ab6483-77d4-4811-aa2a-7d9ee59f18a7" (UID: "e8ab6483-77d4-4811-aa2a-7d9ee59f18a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:04.955965 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.955925    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-kube-api-access-lrst4" (OuterVolumeSpecName: "kube-api-access-lrst4") pod "e8ab6483-77d4-4811-aa2a-7d9ee59f18a7" (UID: "e8ab6483-77d4-4811-aa2a-7d9ee59f18a7"). InnerVolumeSpecName "kube-api-access-lrst4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:04.956040 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:04.956009    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1980f758-2442-4e7d-91f5-52d2a262630a-kube-api-access-wl5hk" (OuterVolumeSpecName: "kube-api-access-wl5hk") pod "1980f758-2442-4e7d-91f5-52d2a262630a" (UID: "1980f758-2442-4e7d-91f5-52d2a262630a"). InnerVolumeSpecName "kube-api-access-wl5hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:05.043625 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.043524    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:05.055653 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.055626    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-maas-api-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:05.055653 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.055651    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl5hk\" (UniqueName: \"kubernetes.io/projected/1980f758-2442-4e7d-91f5-52d2a262630a-kube-api-access-wl5hk\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:05.055804 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.055661    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-openshift-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:05.055804 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.055670    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-metrics-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:05.055804 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.055680    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpr46\" (UniqueName: \"kubernetes.io/projected/4beeb9ee-ace8-41d1-98e1-d227a168f7b5-kube-api-access-tpr46\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:05.055804 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.055688    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrst4\" (UniqueName: \"kubernetes.io/projected/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7-kube-api-access-lrst4\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:05.123103 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.123077    2574 generic.go:356] "Generic (PLEG): container finished" podID="e8ab6483-77d4-4811-aa2a-7d9ee59f18a7" containerID="1f82e83bd0f9987ddeb52e381674a6bd59736690356b0ced8139dbd611ba1990" exitCode=0
Sep 11 22:18:05.123289 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.123117    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp" event={"ID":"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7","Type":"ContainerDied","Data":"1f82e83bd0f9987ddeb52e381674a6bd59736690356b0ced8139dbd611ba1990"}
Sep 11 22:18:05.123412 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.123152    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"
Sep 11 22:18:05.123412 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.123394    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp" event={"ID":"e8ab6483-77d4-4811-aa2a-7d9ee59f18a7","Type":"ContainerDied","Data":"916ce2bf1797f8087f1bc422d9bbe57b7faf3490aecc3c993614fddeb36a1745"}
Sep 11 22:18:05.123523 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.123420    2574 scope.go:117] "RemoveContainer" containerID="1f82e83bd0f9987ddeb52e381674a6bd59736690356b0ced8139dbd611ba1990"
Sep 11 22:18:05.125218 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.125195    2574 generic.go:356] "Generic (PLEG): container finished" podID="4beeb9ee-ace8-41d1-98e1-d227a168f7b5" containerID="6c9fe08f5a396b266df50f3df526733fc6bb887958ef9228762fd236bd378a54" exitCode=0
Sep 11 22:18:05.125359 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.125255    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"
Sep 11 22:18:05.125359 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.125333    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn" event={"ID":"4beeb9ee-ace8-41d1-98e1-d227a168f7b5","Type":"ContainerDied","Data":"6c9fe08f5a396b266df50f3df526733fc6bb887958ef9228762fd236bd378a54"}
Sep 11 22:18:05.125481 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.125373    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn" event={"ID":"4beeb9ee-ace8-41d1-98e1-d227a168f7b5","Type":"ContainerDied","Data":"d7905a5590b94f23a1f8c2144ec99fdea0af246439ede9e055ce15f388d82dcb"}
Sep 11 22:18:05.129496 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.129471    2574 generic.go:356] "Generic (PLEG): container finished" podID="1980f758-2442-4e7d-91f5-52d2a262630a" containerID="3687ce4476cd43cd8ecc05107b2aaab9f79205219893b1425db48285277eef4a" exitCode=0
Sep 11 22:18:05.129603 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.129513    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq" event={"ID":"1980f758-2442-4e7d-91f5-52d2a262630a","Type":"ContainerDied","Data":"3687ce4476cd43cd8ecc05107b2aaab9f79205219893b1425db48285277eef4a"}
Sep 11 22:18:05.129603 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.129520    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"
Sep 11 22:18:05.129603 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.129552    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq" event={"ID":"1980f758-2442-4e7d-91f5-52d2a262630a","Type":"ContainerDied","Data":"665e96b7d7931dbe61a0123a29c40ff548a944edfdd2888fb60e4838c9c6d437"}
Sep 11 22:18:05.135791 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.135767    2574 scope.go:117] "RemoveContainer" containerID="1f82e83bd0f9987ddeb52e381674a6bd59736690356b0ced8139dbd611ba1990"
Sep 11 22:18:05.136252 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:18:05.136228    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f82e83bd0f9987ddeb52e381674a6bd59736690356b0ced8139dbd611ba1990\": container with ID starting with 1f82e83bd0f9987ddeb52e381674a6bd59736690356b0ced8139dbd611ba1990 not found: ID does not exist" containerID="1f82e83bd0f9987ddeb52e381674a6bd59736690356b0ced8139dbd611ba1990"
Sep 11 22:18:05.136378 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.136261    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f82e83bd0f9987ddeb52e381674a6bd59736690356b0ced8139dbd611ba1990"} err="failed to get container status \"1f82e83bd0f9987ddeb52e381674a6bd59736690356b0ced8139dbd611ba1990\": rpc error: code = NotFound desc = could not find container \"1f82e83bd0f9987ddeb52e381674a6bd59736690356b0ced8139dbd611ba1990\": container with ID starting with 1f82e83bd0f9987ddeb52e381674a6bd59736690356b0ced8139dbd611ba1990 not found: ID does not exist"
Sep 11 22:18:05.136378 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.136283    2574 scope.go:117] "RemoveContainer" containerID="6c9fe08f5a396b266df50f3df526733fc6bb887958ef9228762fd236bd378a54"
Sep 11 22:18:05.146671 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.146646    2574 scope.go:117] "RemoveContainer" containerID="6c9fe08f5a396b266df50f3df526733fc6bb887958ef9228762fd236bd378a54"
Sep 11 22:18:05.147047 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:18:05.147019    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9fe08f5a396b266df50f3df526733fc6bb887958ef9228762fd236bd378a54\": container with ID starting with 6c9fe08f5a396b266df50f3df526733fc6bb887958ef9228762fd236bd378a54 not found: ID does not exist" containerID="6c9fe08f5a396b266df50f3df526733fc6bb887958ef9228762fd236bd378a54"
Sep 11 22:18:05.147145 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.147053    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9fe08f5a396b266df50f3df526733fc6bb887958ef9228762fd236bd378a54"} err="failed to get container status \"6c9fe08f5a396b266df50f3df526733fc6bb887958ef9228762fd236bd378a54\": rpc error: code = NotFound desc = could not find container \"6c9fe08f5a396b266df50f3df526733fc6bb887958ef9228762fd236bd378a54\": container with ID starting with 6c9fe08f5a396b266df50f3df526733fc6bb887958ef9228762fd236bd378a54 not found: ID does not exist"
Sep 11 22:18:05.147145 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.147075    2574 scope.go:117] "RemoveContainer" containerID="3687ce4476cd43cd8ecc05107b2aaab9f79205219893b1425db48285277eef4a"
Sep 11 22:18:05.149553 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.149532    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"]
Sep 11 22:18:05.153647 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.153621    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2715d7-579cc74894-zcjxp"]
Sep 11 22:18:05.156518 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.156494    2574 scope.go:117] "RemoveContainer" containerID="3687ce4476cd43cd8ecc05107b2aaab9f79205219893b1425db48285277eef4a"
Sep 11 22:18:05.156805 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:18:05.156784    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3687ce4476cd43cd8ecc05107b2aaab9f79205219893b1425db48285277eef4a\": container with ID starting with 3687ce4476cd43cd8ecc05107b2aaab9f79205219893b1425db48285277eef4a not found: ID does not exist" containerID="3687ce4476cd43cd8ecc05107b2aaab9f79205219893b1425db48285277eef4a"
Sep 11 22:18:05.156885 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.156809    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3687ce4476cd43cd8ecc05107b2aaab9f79205219893b1425db48285277eef4a"} err="failed to get container status \"3687ce4476cd43cd8ecc05107b2aaab9f79205219893b1425db48285277eef4a\": rpc error: code = NotFound desc = could not find container \"3687ce4476cd43cd8ecc05107b2aaab9f79205219893b1425db48285277eef4a\": container with ID starting with 3687ce4476cd43cd8ecc05107b2aaab9f79205219893b1425db48285277eef4a not found: ID does not exist"
Sep 11 22:18:05.163837 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.163814    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"]
Sep 11 22:18:05.167495 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:18:05.167456    2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ab6483_77d4_4811_aa2a_7d9ee59f18a7.slice/crio-916ce2bf1797f8087f1bc422d9bbe57b7faf3490aecc3c993614fddeb36a1745\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1980f758_2442_4e7d_91f5_52d2a262630a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ab6483_77d4_4811_aa2a_7d9ee59f18a7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1980f758_2442_4e7d_91f5_52d2a262630a.slice/crio-665e96b7d7931dbe61a0123a29c40ff548a944edfdd2888fb60e4838c9c6d437\": RecentStats: unable to find data in memory cache]"
Sep 11 22:18:05.170033 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.170014    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2715d7-76df7b8d57-mt4dn"]
Sep 11 22:18:05.179559 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.179537    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"]
Sep 11 22:18:05.182844 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.182818    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2715d7-5d6489d854-ptjvq"]
Sep 11 22:18:05.199322 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.199288    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:05.721416 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.721382    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1980f758-2442-4e7d-91f5-52d2a262630a" path="/var/lib/kubelet/pods/1980f758-2442-4e7d-91f5-52d2a262630a/volumes"
Sep 11 22:18:05.721773 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.721765    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4beeb9ee-ace8-41d1-98e1-d227a168f7b5" path="/var/lib/kubelet/pods/4beeb9ee-ace8-41d1-98e1-d227a168f7b5/volumes"
Sep 11 22:18:05.722107 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.722094    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8ab6483-77d4-4811-aa2a-7d9ee59f18a7" path="/var/lib/kubelet/pods/e8ab6483-77d4-4811-aa2a-7d9ee59f18a7/volumes"
Sep 11 22:18:05.987916 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:05.987868    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:06.035188 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.035152    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"]
Sep 11 22:18:06.049656 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.049615    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t" podUID="7c9bfb9c-29cd-4b02-805b-35cdd9897df4" containerName="payload-processing" containerID="cri-o://5c2ba16a9d261fa53b684b0ff60db177ccdf6b4e55bf325cf6f12eaf72ce1efc" gracePeriod=30
Sep 11 22:18:06.051565 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.051325    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"]
Sep 11 22:18:06.051565 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.051517    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml" podUID="8de5e2bd-d55f-4ef5-9cf4-58d36c4964af" containerName="payload-pre-processing" containerID="cri-o://5dd9412890d50635fa71df5b3be7b8c90764f2b9f7eca9b22052e347182e5a99" gracePeriod=30
Sep 11 22:18:06.077348 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.077297    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:06.140457 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.140427    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:06.241364 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.241268    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:06.321111 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.321082    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:06.341409 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.341389    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"
Sep 11 22:18:06.344404 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.344386    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"
Sep 11 22:18:06.467579 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.467547    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfrh2\" (UniqueName: \"kubernetes.io/projected/8de5e2bd-d55f-4ef5-9cf4-58d36c4964af-kube-api-access-kfrh2\") pod \"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af\" (UID: \"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af\") "
Sep 11 22:18:06.467783 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.467613    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msjb8\" (UniqueName: \"kubernetes.io/projected/7c9bfb9c-29cd-4b02-805b-35cdd9897df4-kube-api-access-msjb8\") pod \"7c9bfb9c-29cd-4b02-805b-35cdd9897df4\" (UID: \"7c9bfb9c-29cd-4b02-805b-35cdd9897df4\") "
Sep 11 22:18:06.467783 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.467640    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de5e2bd-d55f-4ef5-9cf4-58d36c4964af-plugins-config-volume\") pod \"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af\" (UID: \"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af\") "
Sep 11 22:18:06.467783 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.467708    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9bfb9c-29cd-4b02-805b-35cdd9897df4-plugins-config-volume\") pod \"7c9bfb9c-29cd-4b02-805b-35cdd9897df4\" (UID: \"7c9bfb9c-29cd-4b02-805b-35cdd9897df4\") "
Sep 11 22:18:06.468082 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.468059    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de5e2bd-d55f-4ef5-9cf4-58d36c4964af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8de5e2bd-d55f-4ef5-9cf4-58d36c4964af" (UID: "8de5e2bd-d55f-4ef5-9cf4-58d36c4964af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:06.468156 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.468132    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9bfb9c-29cd-4b02-805b-35cdd9897df4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c9bfb9c-29cd-4b02-805b-35cdd9897df4" (UID: "7c9bfb9c-29cd-4b02-805b-35cdd9897df4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:06.469736 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.469712    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de5e2bd-d55f-4ef5-9cf4-58d36c4964af-kube-api-access-kfrh2" (OuterVolumeSpecName: "kube-api-access-kfrh2") pod "8de5e2bd-d55f-4ef5-9cf4-58d36c4964af" (UID: "8de5e2bd-d55f-4ef5-9cf4-58d36c4964af"). InnerVolumeSpecName "kube-api-access-kfrh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:06.469838 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.469788    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9bfb9c-29cd-4b02-805b-35cdd9897df4-kube-api-access-msjb8" (OuterVolumeSpecName: "kube-api-access-msjb8") pod "7c9bfb9c-29cd-4b02-805b-35cdd9897df4" (UID: "7c9bfb9c-29cd-4b02-805b-35cdd9897df4"). InnerVolumeSpecName "kube-api-access-msjb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:06.568729 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.568695    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msjb8\" (UniqueName: \"kubernetes.io/projected/7c9bfb9c-29cd-4b02-805b-35cdd9897df4-kube-api-access-msjb8\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:06.568729 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.568724    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de5e2bd-d55f-4ef5-9cf4-58d36c4964af-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:06.568729 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.568733    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9bfb9c-29cd-4b02-805b-35cdd9897df4-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:06.568924 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.568742    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfrh2\" (UniqueName: \"kubernetes.io/projected/8de5e2bd-d55f-4ef5-9cf4-58d36c4964af-kube-api-access-kfrh2\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:06.617651 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:06.617614    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:07.022729 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.022691    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:07.144490 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.144457    2574 generic.go:356] "Generic (PLEG): container finished" podID="7c9bfb9c-29cd-4b02-805b-35cdd9897df4" containerID="5c2ba16a9d261fa53b684b0ff60db177ccdf6b4e55bf325cf6f12eaf72ce1efc" exitCode=0
Sep 11 22:18:07.144733 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.144509    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"
Sep 11 22:18:07.144733 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.144544    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t" event={"ID":"7c9bfb9c-29cd-4b02-805b-35cdd9897df4","Type":"ContainerDied","Data":"5c2ba16a9d261fa53b684b0ff60db177ccdf6b4e55bf325cf6f12eaf72ce1efc"}
Sep 11 22:18:07.144733 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.144599    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t" event={"ID":"7c9bfb9c-29cd-4b02-805b-35cdd9897df4","Type":"ContainerDied","Data":"3e67ae4a8cdc4ddc069174f450702a6158c886e9f4b76d7e379292d64dba6925"}
Sep 11 22:18:07.144733 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.144625    2574 scope.go:117] "RemoveContainer" containerID="5c2ba16a9d261fa53b684b0ff60db177ccdf6b4e55bf325cf6f12eaf72ce1efc"
Sep 11 22:18:07.145927 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.145884    2574 generic.go:356] "Generic (PLEG): container finished" podID="8de5e2bd-d55f-4ef5-9cf4-58d36c4964af" containerID="5dd9412890d50635fa71df5b3be7b8c90764f2b9f7eca9b22052e347182e5a99" exitCode=0
Sep 11 22:18:07.146009 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.145933    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"
Sep 11 22:18:07.146052 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.146013    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml" event={"ID":"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af","Type":"ContainerDied","Data":"5dd9412890d50635fa71df5b3be7b8c90764f2b9f7eca9b22052e347182e5a99"}
Sep 11 22:18:07.146097 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.146079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml" event={"ID":"8de5e2bd-d55f-4ef5-9cf4-58d36c4964af","Type":"ContainerDied","Data":"fc4056913a3f18ad37c75e7c82baa629688d1a980858388ecfc2978f80d9fdba"}
Sep 11 22:18:07.154188 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.154172    2574 scope.go:117] "RemoveContainer" containerID="5c2ba16a9d261fa53b684b0ff60db177ccdf6b4e55bf325cf6f12eaf72ce1efc"
Sep 11 22:18:07.154493 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:18:07.154468    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2ba16a9d261fa53b684b0ff60db177ccdf6b4e55bf325cf6f12eaf72ce1efc\": container with ID starting with 5c2ba16a9d261fa53b684b0ff60db177ccdf6b4e55bf325cf6f12eaf72ce1efc not found: ID does not exist" containerID="5c2ba16a9d261fa53b684b0ff60db177ccdf6b4e55bf325cf6f12eaf72ce1efc"
Sep 11 22:18:07.154566 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.154507    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2ba16a9d261fa53b684b0ff60db177ccdf6b4e55bf325cf6f12eaf72ce1efc"} err="failed to get container status \"5c2ba16a9d261fa53b684b0ff60db177ccdf6b4e55bf325cf6f12eaf72ce1efc\": rpc error: code = NotFound desc = could not find container \"5c2ba16a9d261fa53b684b0ff60db177ccdf6b4e55bf325cf6f12eaf72ce1efc\": container with ID starting with 5c2ba16a9d261fa53b684b0ff60db177ccdf6b4e55bf325cf6f12eaf72ce1efc not found: ID does not exist"
Sep 11 22:18:07.154566 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.154524    2574 scope.go:117] "RemoveContainer" containerID="5dd9412890d50635fa71df5b3be7b8c90764f2b9f7eca9b22052e347182e5a99"
Sep 11 22:18:07.162429 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.162410    2574 scope.go:117] "RemoveContainer" containerID="5dd9412890d50635fa71df5b3be7b8c90764f2b9f7eca9b22052e347182e5a99"
Sep 11 22:18:07.162708 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:18:07.162691    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd9412890d50635fa71df5b3be7b8c90764f2b9f7eca9b22052e347182e5a99\": container with ID starting with 5dd9412890d50635fa71df5b3be7b8c90764f2b9f7eca9b22052e347182e5a99 not found: ID does not exist" containerID="5dd9412890d50635fa71df5b3be7b8c90764f2b9f7eca9b22052e347182e5a99"
Sep 11 22:18:07.162759 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.162715    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd9412890d50635fa71df5b3be7b8c90764f2b9f7eca9b22052e347182e5a99"} err="failed to get container status \"5dd9412890d50635fa71df5b3be7b8c90764f2b9f7eca9b22052e347182e5a99\": rpc error: code = NotFound desc = could not find container \"5dd9412890d50635fa71df5b3be7b8c90764f2b9f7eca9b22052e347182e5a99\": container with ID starting with 5dd9412890d50635fa71df5b3be7b8c90764f2b9f7eca9b22052e347182e5a99 not found: ID does not exist"
Sep 11 22:18:07.169937 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.169916    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"]
Sep 11 22:18:07.175435 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.175414    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7e2add-56bb4bd4db-jchml"]
Sep 11 22:18:07.184990 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.184971    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"]
Sep 11 22:18:07.188671 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.188650    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7e2add-7fb4c5c9d9-fdq9t"]
Sep 11 22:18:07.720759 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.720731    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c9bfb9c-29cd-4b02-805b-35cdd9897df4" path="/var/lib/kubelet/pods/7c9bfb9c-29cd-4b02-805b-35cdd9897df4/volumes"
Sep 11 22:18:07.721092 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.721078    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8de5e2bd-d55f-4ef5-9cf4-58d36c4964af" path="/var/lib/kubelet/pods/8de5e2bd-d55f-4ef5-9cf4-58d36c4964af/volumes"
Sep 11 22:18:07.839149 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:07.839122    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:08.628452 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:08.628412    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:09.312352 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:09.312292    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:09.967415 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:09.967379    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:10.985635 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:10.985600    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:11.085432 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:11.085396    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:11.240865 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:11.240785    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:12.394045 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:12.394010    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:12.449424 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:12.449388    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:12.605473 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:12.605437    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:12.677210 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:12.677122    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:13.251971 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.251938    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq" podUID="847ba169-10a6-453f-9cd2-ca9ea2dfe24d" containerName="main" containerID="cri-o://4e81acc3b8ab8ac4335a8ff53c3020641e970eb9c2e187d1052dc0c34cd7428f" gracePeriod=45
Sep 11 22:18:13.393520 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.393482    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:13.506264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.506197    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:18:13.633524 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.633493    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-tls-certs\") pod \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") "
Sep 11 22:18:13.633524 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.633526    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-dshm\") pod \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") "
Sep 11 22:18:13.633765 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.633573    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-model-cache\") pod \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") "
Sep 11 22:18:13.633765 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.633601    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdzjm\" (UniqueName: \"kubernetes.io/projected/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-kube-api-access-kdzjm\") pod \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") "
Sep 11 22:18:13.633765 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.633625    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-home\") pod \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") "
Sep 11 22:18:13.633765 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.633689    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-tmp-dir\") pod \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\" (UID: \"847ba169-10a6-453f-9cd2-ca9ea2dfe24d\") "
Sep 11 22:18:13.633979 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.633820    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-model-cache" (OuterVolumeSpecName: "model-cache") pod "847ba169-10a6-453f-9cd2-ca9ea2dfe24d" (UID: "847ba169-10a6-453f-9cd2-ca9ea2dfe24d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.634060 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.634028    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-model-cache\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.634120 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.634046    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "847ba169-10a6-453f-9cd2-ca9ea2dfe24d" (UID: "847ba169-10a6-453f-9cd2-ca9ea2dfe24d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.634120 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.634077    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-home" (OuterVolumeSpecName: "home") pod "847ba169-10a6-453f-9cd2-ca9ea2dfe24d" (UID: "847ba169-10a6-453f-9cd2-ca9ea2dfe24d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.636034 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.636007    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "847ba169-10a6-453f-9cd2-ca9ea2dfe24d" (UID: "847ba169-10a6-453f-9cd2-ca9ea2dfe24d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:13.636274 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.636254    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-dshm" (OuterVolumeSpecName: "dshm") pod "847ba169-10a6-453f-9cd2-ca9ea2dfe24d" (UID: "847ba169-10a6-453f-9cd2-ca9ea2dfe24d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.636371 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.636275    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-kube-api-access-kdzjm" (OuterVolumeSpecName: "kube-api-access-kdzjm") pod "847ba169-10a6-453f-9cd2-ca9ea2dfe24d" (UID: "847ba169-10a6-453f-9cd2-ca9ea2dfe24d"). InnerVolumeSpecName "kube-api-access-kdzjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:13.734557 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.734533    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdzjm\" (UniqueName: \"kubernetes.io/projected/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-kube-api-access-kdzjm\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.734557 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.734556    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-home\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.734708 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.734568    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-tmp-dir\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.734708 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.734577    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-tls-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.734708 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.734584    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ba169-10a6-453f-9cd2-ca9ea2dfe24d-dshm\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.967951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:13.967922    2574 scope.go:117] "RemoveContainer" containerID="4e81acc3b8ab8ac4335a8ff53c3020641e970eb9c2e187d1052dc0c34cd7428f"
Sep 11 22:18:14.185997 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:14.185960    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq" event={"ID":"847ba169-10a6-453f-9cd2-ca9ea2dfe24d","Type":"ContainerDied","Data":"4e81acc3b8ab8ac4335a8ff53c3020641e970eb9c2e187d1052dc0c34cd7428f"}
Sep 11 22:18:14.186161 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:14.186005    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq" event={"ID":"847ba169-10a6-453f-9cd2-ca9ea2dfe24d","Type":"ContainerDied","Data":"e230dea70bf7e779123fc87a8575c9a3df049ffd94f73feba98b978bde31f88b"}
Sep 11 22:18:14.186161 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:14.186012    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"
Sep 11 22:18:14.198543 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:14.198516    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:14.217298 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:14.217264    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"]
Sep 11 22:18:14.222320 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:14.222250    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/premium-w4-7e2add-kserve-7f5f4c877f-6mqbq"]
Sep 11 22:18:14.974115 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:14.974077    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:15.636779 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:15.636741    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:15.721803 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:15.721764    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="847ba169-10a6-453f-9cd2-ca9ea2dfe24d" path="/var/lib/kubelet/pods/847ba169-10a6-453f-9cd2-ca9ea2dfe24d/volumes"
Sep 11 22:18:17.213511 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:17.213478    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:22.194392 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.194356    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"]
Sep 11 22:18:22.194834 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.194556    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg" podUID="e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0" containerName="maas-api" containerID="cri-o://ba05c855adb2392864dc792432d784b73b2e701e72beeb010ca2a63997fe120e" gracePeriod=30
Sep 11 22:18:22.219803 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.219745    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:22.238433 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.238405    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"]
Sep 11 22:18:22.238630 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.238607    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4" podUID="b7185e0d-5978-4e82-83d1-ced33de8c4a4" containerName="payload-processing" containerID="cri-o://63c9b8b41e905d0dafcfcd40227ea5036d3255cedc20733db2a1c2d6a110e888" gracePeriod=30
Sep 11 22:18:22.246036 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.245976    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"]
Sep 11 22:18:22.246324 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.246278    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh" podUID="26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c" containerName="payload-pre-processing" containerID="cri-o://576af2da08e39daa04c63b8e2b7dacb16c0cb9f1e348198325a56c26a3b380d4" gracePeriod=30
Sep 11 22:18:22.282970 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.282938    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:22.345246 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.345221    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:22.390863 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.390688    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh" podUID="26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.64:9004: connect: connection refused"
Sep 11 22:18:22.390863 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.390755    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4" podUID="b7185e0d-5978-4e82-83d1-ced33de8c4a4" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.65:9004: connect: connection refused"
Sep 11 22:18:22.446129 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.446043    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:22.515145 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.515074    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"
Sep 11 22:18:22.538155 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.538131    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:18:22.550489 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.550468    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"
Sep 11 22:18:22.615914 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.615887    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7185e0d-5978-4e82-83d1-ced33de8c4a4-plugins-config-volume\") pod \"b7185e0d-5978-4e82-83d1-ced33de8c4a4\" (UID: \"b7185e0d-5978-4e82-83d1-ced33de8c4a4\") "
Sep 11 22:18:22.616107 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.615942    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn2f2\" (UniqueName: \"kubernetes.io/projected/b7185e0d-5978-4e82-83d1-ced33de8c4a4-kube-api-access-zn2f2\") pod \"b7185e0d-5978-4e82-83d1-ced33de8c4a4\" (UID: \"b7185e0d-5978-4e82-83d1-ced33de8c4a4\") "
Sep 11 22:18:22.616338 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.616290    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7185e0d-5978-4e82-83d1-ced33de8c4a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7185e0d-5978-4e82-83d1-ced33de8c4a4" (UID: "b7185e0d-5978-4e82-83d1-ced33de8c4a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:22.618037 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.618014    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7185e0d-5978-4e82-83d1-ced33de8c4a4-kube-api-access-zn2f2" (OuterVolumeSpecName: "kube-api-access-zn2f2") pod "b7185e0d-5978-4e82-83d1-ced33de8c4a4" (UID: "b7185e0d-5978-4e82-83d1-ced33de8c4a4"). InnerVolumeSpecName "kube-api-access-zn2f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:22.716713 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.716625    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-maas-api-tls\") pod \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") "
Sep 11 22:18:22.716713 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.716665    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-metrics-certs\") pod \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") "
Sep 11 22:18:22.716920 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.716717    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h7vl\" (UniqueName: \"kubernetes.io/projected/26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c-kube-api-access-2h7vl\") pod \"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c\" (UID: \"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c\") "
Sep 11 22:18:22.716920 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.716826    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c-plugins-config-volume\") pod \"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c\" (UID: \"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c\") "
Sep 11 22:18:22.716920 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.716868    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-openshift-service-ca\") pod \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") "
Sep 11 22:18:22.717078 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.716969    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2jkx\" (UniqueName: \"kubernetes.io/projected/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-kube-api-access-s2jkx\") pod \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\" (UID: \"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0\") "
Sep 11 22:18:22.717348 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.717274    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c" (UID: "26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:22.717348 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.717289    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7185e0d-5978-4e82-83d1-ced33de8c4a4-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:22.717503 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.717353    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn2f2\" (UniqueName: \"kubernetes.io/projected/b7185e0d-5978-4e82-83d1-ced33de8c4a4-kube-api-access-zn2f2\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:22.717633 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.717590    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0" (UID: "e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:22.718950 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.718929    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0" (UID: "e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:22.719039 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.718983    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c-kube-api-access-2h7vl" (OuterVolumeSpecName: "kube-api-access-2h7vl") pod "26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c" (UID: "26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c"). InnerVolumeSpecName "kube-api-access-2h7vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:22.719469 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.719447    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-kube-api-access-s2jkx" (OuterVolumeSpecName: "kube-api-access-s2jkx") pod "e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0" (UID: "e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0"). InnerVolumeSpecName "kube-api-access-s2jkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:22.719735 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.719720    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0" (UID: "e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:22.764798 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.764778    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:22.818402 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.818368    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2jkx\" (UniqueName: \"kubernetes.io/projected/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-kube-api-access-s2jkx\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:22.818402 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.818400    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-maas-api-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:22.818548 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.818415    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-metrics-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:22.818548 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.818428    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h7vl\" (UniqueName: \"kubernetes.io/projected/26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c-kube-api-access-2h7vl\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:22.818548 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.818441    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:22.818548 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.818454    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0-openshift-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:22.916469 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:22.916426    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:23.234758 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.234722    2574 generic.go:356] "Generic (PLEG): container finished" podID="26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c" containerID="576af2da08e39daa04c63b8e2b7dacb16c0cb9f1e348198325a56c26a3b380d4" exitCode=0
Sep 11 22:18:23.235167 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.234764    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"
Sep 11 22:18:23.235167 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.234802    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh" event={"ID":"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c","Type":"ContainerDied","Data":"576af2da08e39daa04c63b8e2b7dacb16c0cb9f1e348198325a56c26a3b380d4"}
Sep 11 22:18:23.235167 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.234853    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh" event={"ID":"26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c","Type":"ContainerDied","Data":"46e074943958bc4c5808015814c481f39e6d0801d41c4267799c52c1cc59f0ed"}
Sep 11 22:18:23.235167 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.234878    2574 scope.go:117] "RemoveContainer" containerID="576af2da08e39daa04c63b8e2b7dacb16c0cb9f1e348198325a56c26a3b380d4"
Sep 11 22:18:23.236289 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.236264    2574 generic.go:356] "Generic (PLEG): container finished" podID="e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0" containerID="ba05c855adb2392864dc792432d784b73b2e701e72beeb010ca2a63997fe120e" exitCode=0
Sep 11 22:18:23.236448 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.236328    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"
Sep 11 22:18:23.236448 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.236346    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg" event={"ID":"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0","Type":"ContainerDied","Data":"ba05c855adb2392864dc792432d784b73b2e701e72beeb010ca2a63997fe120e"}
Sep 11 22:18:23.236448 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.236396    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg" event={"ID":"e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0","Type":"ContainerDied","Data":"9b4cbed932c719c87c0befef07d46cfd625e0e71ece71a23f8cd322f6750c6d0"}
Sep 11 22:18:23.237970 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.237944    2574 generic.go:356] "Generic (PLEG): container finished" podID="b7185e0d-5978-4e82-83d1-ced33de8c4a4" containerID="63c9b8b41e905d0dafcfcd40227ea5036d3255cedc20733db2a1c2d6a110e888" exitCode=0
Sep 11 22:18:23.238090 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.237991    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"
Sep 11 22:18:23.238090 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.238005    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4" event={"ID":"b7185e0d-5978-4e82-83d1-ced33de8c4a4","Type":"ContainerDied","Data":"63c9b8b41e905d0dafcfcd40227ea5036d3255cedc20733db2a1c2d6a110e888"}
Sep 11 22:18:23.238090 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.238047    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4" event={"ID":"b7185e0d-5978-4e82-83d1-ced33de8c4a4","Type":"ContainerDied","Data":"ae4f80e7122adbcaccb74c08ad990c0757aea2ad7d0beff49ca488c9238a1baa"}
Sep 11 22:18:23.244467 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.244300    2574 scope.go:117] "RemoveContainer" containerID="576af2da08e39daa04c63b8e2b7dacb16c0cb9f1e348198325a56c26a3b380d4"
Sep 11 22:18:23.244845 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:18:23.244816    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576af2da08e39daa04c63b8e2b7dacb16c0cb9f1e348198325a56c26a3b380d4\": container with ID starting with 576af2da08e39daa04c63b8e2b7dacb16c0cb9f1e348198325a56c26a3b380d4 not found: ID does not exist" containerID="576af2da08e39daa04c63b8e2b7dacb16c0cb9f1e348198325a56c26a3b380d4"
Sep 11 22:18:23.245071 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.244859    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576af2da08e39daa04c63b8e2b7dacb16c0cb9f1e348198325a56c26a3b380d4"} err="failed to get container status \"576af2da08e39daa04c63b8e2b7dacb16c0cb9f1e348198325a56c26a3b380d4\": rpc error: code = NotFound desc = could not find container \"576af2da08e39daa04c63b8e2b7dacb16c0cb9f1e348198325a56c26a3b380d4\": container with ID starting with 576af2da08e39daa04c63b8e2b7dacb16c0cb9f1e348198325a56c26a3b380d4 not found: ID does not exist"
Sep 11 22:18:23.245071 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.244884    2574 scope.go:117] "RemoveContainer" containerID="ba05c855adb2392864dc792432d784b73b2e701e72beeb010ca2a63997fe120e"
Sep 11 22:18:23.253095 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.253076    2574 scope.go:117] "RemoveContainer" containerID="ba05c855adb2392864dc792432d784b73b2e701e72beeb010ca2a63997fe120e"
Sep 11 22:18:23.253360 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:18:23.253343    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba05c855adb2392864dc792432d784b73b2e701e72beeb010ca2a63997fe120e\": container with ID starting with ba05c855adb2392864dc792432d784b73b2e701e72beeb010ca2a63997fe120e not found: ID does not exist" containerID="ba05c855adb2392864dc792432d784b73b2e701e72beeb010ca2a63997fe120e"
Sep 11 22:18:23.253430 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.253365    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba05c855adb2392864dc792432d784b73b2e701e72beeb010ca2a63997fe120e"} err="failed to get container status \"ba05c855adb2392864dc792432d784b73b2e701e72beeb010ca2a63997fe120e\": rpc error: code = NotFound desc = could not find container \"ba05c855adb2392864dc792432d784b73b2e701e72beeb010ca2a63997fe120e\": container with ID starting with ba05c855adb2392864dc792432d784b73b2e701e72beeb010ca2a63997fe120e not found: ID does not exist"
Sep 11 22:18:23.253430 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.253378    2574 scope.go:117] "RemoveContainer" containerID="63c9b8b41e905d0dafcfcd40227ea5036d3255cedc20733db2a1c2d6a110e888"
Sep 11 22:18:23.261633 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.261620    2574 scope.go:117] "RemoveContainer" containerID="63c9b8b41e905d0dafcfcd40227ea5036d3255cedc20733db2a1c2d6a110e888"
Sep 11 22:18:23.261903 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:18:23.261884    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c9b8b41e905d0dafcfcd40227ea5036d3255cedc20733db2a1c2d6a110e888\": container with ID starting with 63c9b8b41e905d0dafcfcd40227ea5036d3255cedc20733db2a1c2d6a110e888 not found: ID does not exist" containerID="63c9b8b41e905d0dafcfcd40227ea5036d3255cedc20733db2a1c2d6a110e888"
Sep 11 22:18:23.261952 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.261907    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c9b8b41e905d0dafcfcd40227ea5036d3255cedc20733db2a1c2d6a110e888"} err="failed to get container status \"63c9b8b41e905d0dafcfcd40227ea5036d3255cedc20733db2a1c2d6a110e888\": rpc error: code = NotFound desc = could not find container \"63c9b8b41e905d0dafcfcd40227ea5036d3255cedc20733db2a1c2d6a110e888\": container with ID starting with 63c9b8b41e905d0dafcfcd40227ea5036d3255cedc20733db2a1c2d6a110e888 not found: ID does not exist"
Sep 11 22:18:23.262226 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.262208    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"]
Sep 11 22:18:23.265445 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.265425    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5ef59c-5c4d489d78-2vkfh"]
Sep 11 22:18:23.277083 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.277059    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"]
Sep 11 22:18:23.278858 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.278839    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5ef59c-5559d8cb7f-fsqtg"]
Sep 11 22:18:23.289833 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.289813    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"]
Sep 11 22:18:23.293252 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.293232    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5ef59c-65995898f5-mmnl4"]
Sep 11 22:18:23.714039 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.714006    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:23.720499 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.720469    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c" path="/var/lib/kubelet/pods/26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c/volumes"
Sep 11 22:18:23.720960 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.720945    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7185e0d-5978-4e82-83d1-ced33de8c4a4" path="/var/lib/kubelet/pods/b7185e0d-5978-4e82-83d1-ced33de8c4a4/volumes"
Sep 11 22:18:23.721441 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:23.721421    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0" path="/var/lib/kubelet/pods/e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0/volumes"
Sep 11 22:18:24.501786 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:24.501752    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:25.197197 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:25.197158    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:26.489023 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:26.488992    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:26.589693 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:26.589654    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:27.946483 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:27.946442    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:28.747716 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:28.747677    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:29.199410 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:29.199370    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:29.256019 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:29.255986    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:29.446088 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:29.446057    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:30.318099 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:30.318066    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:30.585732 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:30.585706    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:31.426504 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:31.426467    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:32.115981 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:32.115946    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:32.894841 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:32.894805    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:33.879546 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:33.879515    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:37.799695 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.799663    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"]
Sep 11 22:18:37.800060 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800050    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d2f722-f008-40e1-81d2-d7514abd5ad7" containerName="main"
Sep 11 22:18:37.800105 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800062    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d2f722-f008-40e1-81d2-d7514abd5ad7" containerName="main"
Sep 11 22:18:37.800105 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800071    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4beeb9ee-ace8-41d1-98e1-d227a168f7b5" containerName="payload-processing"
Sep 11 22:18:37.800105 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800076    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4beeb9ee-ace8-41d1-98e1-d227a168f7b5" containerName="payload-processing"
Sep 11 22:18:37.800105 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800091    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c" containerName="payload-pre-processing"
Sep 11 22:18:37.800105 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800096    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c" containerName="payload-pre-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800108    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58d6418-db5f-41c3-9908-1547d5f7cdec" containerName="payload-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800114    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58d6418-db5f-41c3-9908-1547d5f7cdec" containerName="payload-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800120    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ab6483-77d4-4811-aa2a-7d9ee59f18a7" containerName="maas-api"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800125    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ab6483-77d4-4811-aa2a-7d9ee59f18a7" containerName="maas-api"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800133    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f436254-0576-490b-9d08-9968d66f3b85" containerName="payload-pre-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800138    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f436254-0576-490b-9d08-9968d66f3b85" containerName="payload-pre-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800145    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847ba169-10a6-453f-9cd2-ca9ea2dfe24d" containerName="main"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800150    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="847ba169-10a6-453f-9cd2-ca9ea2dfe24d" containerName="main"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800160    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0" containerName="maas-api"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800165    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0" containerName="maas-api"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800172    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d291c1a9-5ba8-4c01-8509-9e2c1c911134" containerName="main"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800178    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d291c1a9-5ba8-4c01-8509-9e2c1c911134" containerName="main"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800184    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7185e0d-5978-4e82-83d1-ced33de8c4a4" containerName="payload-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800190    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7185e0d-5978-4e82-83d1-ced33de8c4a4" containerName="payload-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800199    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c9bfb9c-29cd-4b02-805b-35cdd9897df4" containerName="payload-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800204    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c9bfb9c-29cd-4b02-805b-35cdd9897df4" containerName="payload-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800210    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1980f758-2442-4e7d-91f5-52d2a262630a" containerName="payload-pre-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800215    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1980f758-2442-4e7d-91f5-52d2a262630a" containerName="payload-pre-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800222    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de5e2bd-d55f-4ef5-9cf4-58d36c4964af" containerName="payload-pre-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800227    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de5e2bd-d55f-4ef5-9cf4-58d36c4964af" containerName="payload-pre-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800233    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="624908c2-bcdf-4891-b43a-9e1c33247e62" containerName="payload-processing"
Sep 11 22:18:37.800264 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800238    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="624908c2-bcdf-4891-b43a-9e1c33247e62" containerName="payload-processing"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800299    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="26851f6d-0dd2-4dbc-a9a8-9bb62fcf7b0c" containerName="payload-pre-processing"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800321    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d2f722-f008-40e1-81d2-d7514abd5ad7" containerName="main"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800328    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="624908c2-bcdf-4891-b43a-9e1c33247e62" containerName="payload-processing"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800335    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b58d6418-db5f-41c3-9908-1547d5f7cdec" containerName="payload-processing"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800341    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d291c1a9-5ba8-4c01-8509-9e2c1c911134" containerName="main"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800346    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ab6483-77d4-4811-aa2a-7d9ee59f18a7" containerName="maas-api"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800352    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f436254-0576-490b-9d08-9968d66f3b85" containerName="payload-pre-processing"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800357    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7185e0d-5978-4e82-83d1-ced33de8c4a4" containerName="payload-processing"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800361    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c9bfb9c-29cd-4b02-805b-35cdd9897df4" containerName="payload-processing"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800366    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68c80ba-cdf2-43c1-8b91-9cc7c9b9afe0" containerName="maas-api"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800371    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de5e2bd-d55f-4ef5-9cf4-58d36c4964af" containerName="payload-pre-processing"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800379    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="847ba169-10a6-453f-9cd2-ca9ea2dfe24d" containerName="main"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800383    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1980f758-2442-4e7d-91f5-52d2a262630a" containerName="payload-pre-processing"
Sep 11 22:18:37.800951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.800388    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4beeb9ee-ace8-41d1-98e1-d227a168f7b5" containerName="payload-processing"
Sep 11 22:18:37.802812 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.802796    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.805736 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.805712    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d09c96\"/\"openshift-service-ca.crt\""
Sep 11 22:18:37.805881 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.805748    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d09c96\"/\"sub-test-model-d09c96-kserve-self-signed-certs\""
Sep 11 22:18:37.807201 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.807181    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d09c96\"/\"kube-root-ca.crt\""
Sep 11 22:18:37.807361 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.807181    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d09c96\"/\"default-dockercfg-jp7v5\""
Sep 11 22:18:37.814767 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.814742    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"]
Sep 11 22:18:37.827081 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.827054    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:37.842493 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.842462    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79m5j\" (UniqueName: \"kubernetes.io/projected/d009e277-9fd8-4d81-8bfb-f41fb817d61d-kube-api-access-79m5j\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.842622 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.842505    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-tmp-dir\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.842622 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.842609    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d009e277-9fd8-4d81-8bfb-f41fb817d61d-tls-certs\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.842745 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.842651    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-model-cache\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.842745 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.842678    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-dshm\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.842745 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.842740    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-home\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.943965 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.943938    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-home\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.944125 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.944011    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79m5j\" (UniqueName: \"kubernetes.io/projected/d009e277-9fd8-4d81-8bfb-f41fb817d61d-kube-api-access-79m5j\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.944125 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.944028    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-tmp-dir\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.944125 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.944058    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d009e277-9fd8-4d81-8bfb-f41fb817d61d-tls-certs\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.944125 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.944078    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-model-cache\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.944125 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.944095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-dshm\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.944422 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.944401    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-home\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.944462 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.944442    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-model-cache\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.944500 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.944467    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-tmp-dir\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.946368 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.946339    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-dshm\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.946487 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.946444    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d009e277-9fd8-4d81-8bfb-f41fb817d61d-tls-certs\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:37.953145 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:37.953114    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79m5j\" (UniqueName: \"kubernetes.io/projected/d009e277-9fd8-4d81-8bfb-f41fb817d61d-kube-api-access-79m5j\") pod \"sub-test-model-d09c96-kserve-6f89b5c7b5-872gz\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:38.120443 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:38.120412    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:38.414509 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:38.414483    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"]
Sep 11 22:18:38.416079 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:18:38.416050    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd009e277_9fd8_4d81_8bfb_f41fb817d61d.slice/crio-9a4266f8800f6614d0377e8cdf6e03b7c49bc69f23ab72fc0e89030dc1391f53 WatchSource:0}: Error finding container 9a4266f8800f6614d0377e8cdf6e03b7c49bc69f23ab72fc0e89030dc1391f53: Status 404 returned error can't find the container with id 9a4266f8800f6614d0377e8cdf6e03b7c49bc69f23ab72fc0e89030dc1391f53
Sep 11 22:18:39.330616 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:39.330587    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz" event={"ID":"d009e277-9fd8-4d81-8bfb-f41fb817d61d","Type":"ContainerStarted","Data":"ad10c629c6fb81717d7e2dd850b1be9013d88a4779b6105b154c7ad86ca660b0"}
Sep 11 22:18:39.331006 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:39.330624    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz" event={"ID":"d009e277-9fd8-4d81-8bfb-f41fb817d61d","Type":"ContainerStarted","Data":"9a4266f8800f6614d0377e8cdf6e03b7c49bc69f23ab72fc0e89030dc1391f53"}
Sep 11 22:18:39.352853 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:39.352807    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz" podStartSLOduration=2.352791908 podStartE2EDuration="2.352791908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:18:39.351723252 +0000 UTC m=+1586.086834588" watchObservedRunningTime="2026-09-11 22:18:39.352791908 +0000 UTC m=+1586.087903254"
Sep 11 22:18:40.186700 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.186666    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:40.290526 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.290495    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"]
Sep 11 22:18:40.290715 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.290689    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc" podUID="5fadb3d5-8da2-4dde-8dd2-0472a4c073f9" containerName="maas-api" containerID="cri-o://3f72d1d3db2787bf333f6be5b0dc2189e006122d52564e74332e950791472b6c" gracePeriod=30
Sep 11 22:18:40.322907 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.322879    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75"]
Sep 11 22:18:40.324588 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.324549    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:40.325959 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.325938    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-38f772-29819415-vrr75"]
Sep 11 22:18:40.358698 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.358673    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"]
Sep 11 22:18:40.359042 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.358884    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6" podUID="2c696561-1418-46fe-9f78-4c53b3544548" containerName="payload-pre-processing" containerID="cri-o://274b1c59dbf63b9cefa092941c1800a111f0c009bc997efb1c9ba9c9ce33f46b" gracePeriod=30
Sep 11 22:18:40.424245 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.424218    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:40.525539 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.525250    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:40.610205 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.610174    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:18:40.621736 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.621549    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:40.637382 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.637360    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"
Sep 11 22:18:40.667970 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.667939    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm99x\" (UniqueName: \"kubernetes.io/projected/2c696561-1418-46fe-9f78-4c53b3544548-kube-api-access-gm99x\") pod \"2c696561-1418-46fe-9f78-4c53b3544548\" (UID: \"2c696561-1418-46fe-9f78-4c53b3544548\") "
Sep 11 22:18:40.668135 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.668033    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qxd7\" (UniqueName: \"kubernetes.io/projected/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-kube-api-access-6qxd7\") pod \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") "
Sep 11 22:18:40.668135 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.668104    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-openshift-service-ca\") pod \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") "
Sep 11 22:18:40.668135 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.668130    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c696561-1418-46fe-9f78-4c53b3544548-plugins-config-volume\") pod \"2c696561-1418-46fe-9f78-4c53b3544548\" (UID: \"2c696561-1418-46fe-9f78-4c53b3544548\") "
Sep 11 22:18:40.668321 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.668163    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-metrics-certs\") pod \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") "
Sep 11 22:18:40.668321 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.668192    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-maas-api-tls\") pod \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\" (UID: \"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9\") "
Sep 11 22:18:40.668897 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.668690    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c696561-1418-46fe-9f78-4c53b3544548-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c696561-1418-46fe-9f78-4c53b3544548" (UID: "2c696561-1418-46fe-9f78-4c53b3544548"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:40.668897 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.668851    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5fadb3d5-8da2-4dde-8dd2-0472a4c073f9" (UID: "5fadb3d5-8da2-4dde-8dd2-0472a4c073f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:40.670693 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.670670    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c696561-1418-46fe-9f78-4c53b3544548-kube-api-access-gm99x" (OuterVolumeSpecName: "kube-api-access-gm99x") pod "2c696561-1418-46fe-9f78-4c53b3544548" (UID: "2c696561-1418-46fe-9f78-4c53b3544548"). InnerVolumeSpecName "kube-api-access-gm99x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:40.670857 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.670837    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-kube-api-access-6qxd7" (OuterVolumeSpecName: "kube-api-access-6qxd7") pod "5fadb3d5-8da2-4dde-8dd2-0472a4c073f9" (UID: "5fadb3d5-8da2-4dde-8dd2-0472a4c073f9"). InnerVolumeSpecName "kube-api-access-6qxd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:40.670938 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.670838    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5fadb3d5-8da2-4dde-8dd2-0472a4c073f9" (UID: "5fadb3d5-8da2-4dde-8dd2-0472a4c073f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:40.670938 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.670899    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5fadb3d5-8da2-4dde-8dd2-0472a4c073f9" (UID: "5fadb3d5-8da2-4dde-8dd2-0472a4c073f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:40.769147 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.769068    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-openshift-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:40.769147 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.769096    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c696561-1418-46fe-9f78-4c53b3544548-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:40.769147 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.769106    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-metrics-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:40.769147 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.769114    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-maas-api-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:40.769147 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.769125    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm99x\" (UniqueName: \"kubernetes.io/projected/2c696561-1418-46fe-9f78-4c53b3544548-kube-api-access-gm99x\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:40.769147 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.769138    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qxd7\" (UniqueName: \"kubernetes.io/projected/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9-kube-api-access-6qxd7\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:18:40.920910 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:40.920872    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:41.020293 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.020197    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:41.342451 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.342416    2574 generic.go:356] "Generic (PLEG): container finished" podID="5fadb3d5-8da2-4dde-8dd2-0472a4c073f9" containerID="3f72d1d3db2787bf333f6be5b0dc2189e006122d52564e74332e950791472b6c" exitCode=0
Sep 11 22:18:41.342588 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.342467    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"
Sep 11 22:18:41.342588 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.342485    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc" event={"ID":"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9","Type":"ContainerDied","Data":"3f72d1d3db2787bf333f6be5b0dc2189e006122d52564e74332e950791472b6c"}
Sep 11 22:18:41.342588 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.342542    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc" event={"ID":"5fadb3d5-8da2-4dde-8dd2-0472a4c073f9","Type":"ContainerDied","Data":"dfce887381fd140c9b5ac6ca1839b2f748d0c48f766cd84485dd7ad314d38b92"}
Sep 11 22:18:41.342588 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.342570    2574 scope.go:117] "RemoveContainer" containerID="3f72d1d3db2787bf333f6be5b0dc2189e006122d52564e74332e950791472b6c"
Sep 11 22:18:41.344251 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.344030    2574 generic.go:356] "Generic (PLEG): container finished" podID="2c696561-1418-46fe-9f78-4c53b3544548" containerID="274b1c59dbf63b9cefa092941c1800a111f0c009bc997efb1c9ba9c9ce33f46b" exitCode=0
Sep 11 22:18:41.344251 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.344072    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"
Sep 11 22:18:41.344251 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.344078    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6" event={"ID":"2c696561-1418-46fe-9f78-4c53b3544548","Type":"ContainerDied","Data":"274b1c59dbf63b9cefa092941c1800a111f0c009bc997efb1c9ba9c9ce33f46b"}
Sep 11 22:18:41.344251 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.344138    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6" event={"ID":"2c696561-1418-46fe-9f78-4c53b3544548","Type":"ContainerDied","Data":"9a9b7f96e32a7a487f9a05b93236514a7f59b22945a0bded889afeed33fec1a0"}
Sep 11 22:18:41.353377 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.353360    2574 scope.go:117] "RemoveContainer" containerID="3f72d1d3db2787bf333f6be5b0dc2189e006122d52564e74332e950791472b6c"
Sep 11 22:18:41.353684 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:18:41.353663    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f72d1d3db2787bf333f6be5b0dc2189e006122d52564e74332e950791472b6c\": container with ID starting with 3f72d1d3db2787bf333f6be5b0dc2189e006122d52564e74332e950791472b6c not found: ID does not exist" containerID="3f72d1d3db2787bf333f6be5b0dc2189e006122d52564e74332e950791472b6c"
Sep 11 22:18:41.353782 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.353690    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f72d1d3db2787bf333f6be5b0dc2189e006122d52564e74332e950791472b6c"} err="failed to get container status \"3f72d1d3db2787bf333f6be5b0dc2189e006122d52564e74332e950791472b6c\": rpc error: code = NotFound desc = could not find container \"3f72d1d3db2787bf333f6be5b0dc2189e006122d52564e74332e950791472b6c\": container with ID starting with 3f72d1d3db2787bf333f6be5b0dc2189e006122d52564e74332e950791472b6c not found: ID does not exist"
Sep 11 22:18:41.353782 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.353705    2574 scope.go:117] "RemoveContainer" containerID="274b1c59dbf63b9cefa092941c1800a111f0c009bc997efb1c9ba9c9ce33f46b"
Sep 11 22:18:41.361712 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.361696    2574 scope.go:117] "RemoveContainer" containerID="274b1c59dbf63b9cefa092941c1800a111f0c009bc997efb1c9ba9c9ce33f46b"
Sep 11 22:18:41.361990 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:18:41.361930    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274b1c59dbf63b9cefa092941c1800a111f0c009bc997efb1c9ba9c9ce33f46b\": container with ID starting with 274b1c59dbf63b9cefa092941c1800a111f0c009bc997efb1c9ba9c9ce33f46b not found: ID does not exist" containerID="274b1c59dbf63b9cefa092941c1800a111f0c009bc997efb1c9ba9c9ce33f46b"
Sep 11 22:18:41.361990 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.361956    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274b1c59dbf63b9cefa092941c1800a111f0c009bc997efb1c9ba9c9ce33f46b"} err="failed to get container status \"274b1c59dbf63b9cefa092941c1800a111f0c009bc997efb1c9ba9c9ce33f46b\": rpc error: code = NotFound desc = could not find container \"274b1c59dbf63b9cefa092941c1800a111f0c009bc997efb1c9ba9c9ce33f46b\": container with ID starting with 274b1c59dbf63b9cefa092941c1800a111f0c009bc997efb1c9ba9c9ce33f46b not found: ID does not exist"
Sep 11 22:18:41.369865 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.369844    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"]
Sep 11 22:18:41.373897 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.373877    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6"]
Sep 11 22:18:41.384525 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.384507    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"]
Sep 11 22:18:41.392536 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.392518    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc"]
Sep 11 22:18:41.564561 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.564475    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-38f772-5f578874fd-sgcx6" podUID="2c696561-1418-46fe-9f78-4c53b3544548" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.67:9004: i/o timeout"
Sep 11 22:18:41.720371 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.720339    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c696561-1418-46fe-9f78-4c53b3544548" path="/var/lib/kubelet/pods/2c696561-1418-46fe-9f78-4c53b3544548/volumes"
Sep 11 22:18:41.720838 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.720814    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fadb3d5-8da2-4dde-8dd2-0472a4c073f9" path="/var/lib/kubelet/pods/5fadb3d5-8da2-4dde-8dd2-0472a4c073f9/volumes"
Sep 11 22:18:41.721142 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.721124    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2110600-2cd7-440c-8b8f-ea6aece3be2b" path="/var/lib/kubelet/pods/c2110600-2cd7-440c-8b8f-ea6aece3be2b/volumes"
Sep 11 22:18:41.774890 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:41.774860    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:42.495197 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:42.495158    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:43.564647 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:43.564605    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-38f772-7d9c5ddd6-lpdxc" podUID="5fadb3d5-8da2-4dde-8dd2-0472a4c073f9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.66:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 11 22:18:46.762509 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:46.762468    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:46.864674 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:46.864645    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:47.076915 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:47.076835    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:18:48.121022 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:48.120981    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:48.121503 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:48.121039    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:48.134270 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:48.134249    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:48.393467 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:48.393393    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:18:54.179292 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:18:54.179252    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:19:09.571147 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:19:09.571064    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:19:16.608512 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:19:16.608479    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:19:59.177349 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:19:59.177299    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"]
Sep 11 22:19:59.210158 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:19:59.210123    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:20:00.183155 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:00.183124    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:20:08.419682 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:08.419643    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:20:14.196013 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.195976    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz" podUID="d009e277-9fd8-4d81-8bfb-f41fb817d61d" containerName="main" containerID="cri-o://ad10c629c6fb81717d7e2dd850b1be9013d88a4779b6105b154c7ad86ca660b0" gracePeriod=45
Sep 11 22:20:14.440856 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.440833    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:20:14.485095 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.485029    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79m5j\" (UniqueName: \"kubernetes.io/projected/d009e277-9fd8-4d81-8bfb-f41fb817d61d-kube-api-access-79m5j\") pod \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") "
Sep 11 22:20:14.485095 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.485076    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-model-cache\") pod \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") "
Sep 11 22:20:14.485267 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.485107    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-tmp-dir\") pod \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") "
Sep 11 22:20:14.485267 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.485136    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-dshm\") pod \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") "
Sep 11 22:20:14.485267 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.485176    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d009e277-9fd8-4d81-8bfb-f41fb817d61d-tls-certs\") pod \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") "
Sep 11 22:20:14.485267 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.485235    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-home\") pod \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\" (UID: \"d009e277-9fd8-4d81-8bfb-f41fb817d61d\") "
Sep 11 22:20:14.485510 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.485368    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d009e277-9fd8-4d81-8bfb-f41fb817d61d" (UID: "d009e277-9fd8-4d81-8bfb-f41fb817d61d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:20:14.485510 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.485421    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d009e277-9fd8-4d81-8bfb-f41fb817d61d" (UID: "d009e277-9fd8-4d81-8bfb-f41fb817d61d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:20:14.485619 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.485599    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-home" (OuterVolumeSpecName: "home") pod "d009e277-9fd8-4d81-8bfb-f41fb817d61d" (UID: "d009e277-9fd8-4d81-8bfb-f41fb817d61d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:20:14.485685 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.485621    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-model-cache\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:20:14.485685 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.485640    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-tmp-dir\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:20:14.487374 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.487353    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-dshm" (OuterVolumeSpecName: "dshm") pod "d009e277-9fd8-4d81-8bfb-f41fb817d61d" (UID: "d009e277-9fd8-4d81-8bfb-f41fb817d61d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:20:14.487472 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.487404    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d009e277-9fd8-4d81-8bfb-f41fb817d61d-kube-api-access-79m5j" (OuterVolumeSpecName: "kube-api-access-79m5j") pod "d009e277-9fd8-4d81-8bfb-f41fb817d61d" (UID: "d009e277-9fd8-4d81-8bfb-f41fb817d61d"). InnerVolumeSpecName "kube-api-access-79m5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:20:14.487472 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.487412    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d009e277-9fd8-4d81-8bfb-f41fb817d61d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d009e277-9fd8-4d81-8bfb-f41fb817d61d" (UID: "d009e277-9fd8-4d81-8bfb-f41fb817d61d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:20:14.586343 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.586297    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79m5j\" (UniqueName: \"kubernetes.io/projected/d009e277-9fd8-4d81-8bfb-f41fb817d61d-kube-api-access-79m5j\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:20:14.586343 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.586340    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-dshm\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:20:14.586480 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.586350    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d009e277-9fd8-4d81-8bfb-f41fb817d61d-tls-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:20:14.586480 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.586359    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d009e277-9fd8-4d81-8bfb-f41fb817d61d-home\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:20:14.821162 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.821078    2574 generic.go:356] "Generic (PLEG): container finished" podID="d009e277-9fd8-4d81-8bfb-f41fb817d61d" containerID="ad10c629c6fb81717d7e2dd850b1be9013d88a4779b6105b154c7ad86ca660b0" exitCode=0
Sep 11 22:20:14.821162 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.821125    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"
Sep 11 22:20:14.821370 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.821176    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz" event={"ID":"d009e277-9fd8-4d81-8bfb-f41fb817d61d","Type":"ContainerDied","Data":"ad10c629c6fb81717d7e2dd850b1be9013d88a4779b6105b154c7ad86ca660b0"}
Sep 11 22:20:14.821370 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.821224    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz" event={"ID":"d009e277-9fd8-4d81-8bfb-f41fb817d61d","Type":"ContainerDied","Data":"9a4266f8800f6614d0377e8cdf6e03b7c49bc69f23ab72fc0e89030dc1391f53"}
Sep 11 22:20:14.821370 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.821246    2574 scope.go:117] "RemoveContainer" containerID="ad10c629c6fb81717d7e2dd850b1be9013d88a4779b6105b154c7ad86ca660b0"
Sep 11 22:20:14.830168 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.830152    2574 scope.go:117] "RemoveContainer" containerID="ad10c629c6fb81717d7e2dd850b1be9013d88a4779b6105b154c7ad86ca660b0"
Sep 11 22:20:14.830443 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:20:14.830421    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad10c629c6fb81717d7e2dd850b1be9013d88a4779b6105b154c7ad86ca660b0\": container with ID starting with ad10c629c6fb81717d7e2dd850b1be9013d88a4779b6105b154c7ad86ca660b0 not found: ID does not exist" containerID="ad10c629c6fb81717d7e2dd850b1be9013d88a4779b6105b154c7ad86ca660b0"
Sep 11 22:20:14.830519 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.830455    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad10c629c6fb81717d7e2dd850b1be9013d88a4779b6105b154c7ad86ca660b0"} err="failed to get container status \"ad10c629c6fb81717d7e2dd850b1be9013d88a4779b6105b154c7ad86ca660b0\": rpc error: code = NotFound desc = could not find container \"ad10c629c6fb81717d7e2dd850b1be9013d88a4779b6105b154c7ad86ca660b0\": container with ID starting with ad10c629c6fb81717d7e2dd850b1be9013d88a4779b6105b154c7ad86ca660b0 not found: ID does not exist"
Sep 11 22:20:14.843888 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.843863    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"]
Sep 11 22:20:14.848168 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:14.848149    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/sub-test-model-d09c96-kserve-6f89b5c7b5-872gz"]
Sep 11 22:20:15.727986 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:15.727953    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d009e277-9fd8-4d81-8bfb-f41fb817d61d" path="/var/lib/kubelet/pods/d009e277-9fd8-4d81-8bfb-f41fb817d61d/volumes"
Sep 11 22:20:41.984190 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.984158    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"]
Sep 11 22:20:41.984719 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.984702    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c696561-1418-46fe-9f78-4c53b3544548" containerName="payload-pre-processing"
Sep 11 22:20:41.984773 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.984722    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c696561-1418-46fe-9f78-4c53b3544548" containerName="payload-pre-processing"
Sep 11 22:20:41.984773 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.984751    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d009e277-9fd8-4d81-8bfb-f41fb817d61d" containerName="main"
Sep 11 22:20:41.984773 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.984757    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d009e277-9fd8-4d81-8bfb-f41fb817d61d" containerName="main"
Sep 11 22:20:41.984773 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.984762    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fadb3d5-8da2-4dde-8dd2-0472a4c073f9" containerName="maas-api"
Sep 11 22:20:41.984773 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.984769    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fadb3d5-8da2-4dde-8dd2-0472a4c073f9" containerName="maas-api"
Sep 11 22:20:41.984962 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.984830    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d009e277-9fd8-4d81-8bfb-f41fb817d61d" containerName="main"
Sep 11 22:20:41.984962 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.984838    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fadb3d5-8da2-4dde-8dd2-0472a4c073f9" containerName="maas-api"
Sep 11 22:20:41.984962 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.984844    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c696561-1418-46fe-9f78-4c53b3544548" containerName="payload-pre-processing"
Sep 11 22:20:41.988161 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.988141    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:41.990939 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.990916    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-966527\"/\"kube-root-ca.crt\""
Sep 11 22:20:41.992228 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.992208    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-966527\"/\"rate-test-model-966527-kserve-self-signed-certs\""
Sep 11 22:20:41.992358 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.992264    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-966527\"/\"openshift-service-ca.crt\""
Sep 11 22:20:41.992432 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.992412    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-966527\"/\"default-dockercfg-tc5dd\""
Sep 11 22:20:41.995433 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:41.995409    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"]
Sep 11 22:20:42.009203 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.009173    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:20:42.122013 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.121982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw4k2\" (UniqueName: \"kubernetes.io/projected/e862dd9d-962c-4b90-926f-04723bac28a7-kube-api-access-kw4k2\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.122013 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.122017    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-model-cache\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.122238 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.122045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-home\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.122238 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.122109    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-tmp-dir\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.122238 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.122156    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-dshm\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.122238 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.122192    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e862dd9d-962c-4b90-926f-04723bac28a7-tls-certs\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.223462 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.223431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e862dd9d-962c-4b90-926f-04723bac28a7-tls-certs\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.223632 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.223487    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw4k2\" (UniqueName: \"kubernetes.io/projected/e862dd9d-962c-4b90-926f-04723bac28a7-kube-api-access-kw4k2\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.223632 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.223603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-model-cache\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.223759 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.223663    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-home\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.223759 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.223702    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-tmp-dir\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.223871 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.223768    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-dshm\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.224039 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.224020    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-model-cache\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.224129 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.224065    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-home\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.224185 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.224144    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-tmp-dir\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.225958 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.225936    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-dshm\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.226297 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.226282    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e862dd9d-962c-4b90-926f-04723bac28a7-tls-certs\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.231656 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.231633    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw4k2\" (UniqueName: \"kubernetes.io/projected/e862dd9d-962c-4b90-926f-04723bac28a7-kube-api-access-kw4k2\") pod \"rate-test-model-966527-kserve-66df9846c9-6ds52\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") " pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.308494 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.308426    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:42.578555 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.578530    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"]
Sep 11 22:20:42.580899 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:20:42.580871    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode862dd9d_962c_4b90_926f_04723bac28a7.slice/crio-7dd38f23359b322a6494c30d81933907cb9e72b1abc769b0a7d8c1923331b13f WatchSource:0}: Error finding container 7dd38f23359b322a6494c30d81933907cb9e72b1abc769b0a7d8c1923331b13f: Status 404 returned error can't find the container with id 7dd38f23359b322a6494c30d81933907cb9e72b1abc769b0a7d8c1923331b13f
Sep 11 22:20:42.961173 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.961135    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52" event={"ID":"e862dd9d-962c-4b90-926f-04723bac28a7","Type":"ContainerStarted","Data":"ff5bbbcf8d15981ca620ac37cb54bea028e00c41b1b1c532589d3084e67ae64e"}
Sep 11 22:20:42.961362 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.961185    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52" event={"ID":"e862dd9d-962c-4b90-926f-04723bac28a7","Type":"ContainerStarted","Data":"7dd38f23359b322a6494c30d81933907cb9e72b1abc769b0a7d8c1923331b13f"}
Sep 11 22:20:42.978048 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:42.977997    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52" podStartSLOduration=1.977983576 podStartE2EDuration="1.977983576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:20:42.976625987 +0000 UTC m=+1709.711737331" watchObservedRunningTime="2026-09-11 22:20:42.977983576 +0000 UTC m=+1709.713094922"
Sep 11 22:20:52.309032 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:52.309004    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:52.309427 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:52.309081    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:52.325768 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:52.325744    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:20:53.024769 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:20:53.024738    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:21:09.565714 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:09.565677    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:09.666608 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:09.666575    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:14.057806 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:14.057777    2574 scope.go:117] "RemoveContainer" containerID="688244c6b5672a5cbd0621f05768e191d6ec532735a7fcc0c4f7cfe65d9e9122"
Sep 11 22:21:14.066187 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:14.066161    2574 scope.go:117] "RemoveContainer" containerID="1d64b1fce00e261c2c29094677ad1fbe33263a7703da59a19b25820d2b7fbf0f"
Sep 11 22:21:19.918896 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:19.918854    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:20.865794 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:20.865760    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"]
Sep 11 22:21:20.884904 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:20.884874    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:21.390674 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:21.390644    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:21.724862 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:21.724784    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:21.955642 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:21.955604    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:24.217377 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:24.217345    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:24.905886 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:24.905851    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:24.999837 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:24.999803    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:25.723812 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:25.723772    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:26.486478 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.486449    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"]
Sep 11 22:21:26.490322 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.490290    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:26.493394 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.493369    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:21:26.493518 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.493460    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 22:21:26.494668 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.494649    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6fe4c2-serving-cert\""
Sep 11 22:21:26.494822 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.494708    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mc9f9\""
Sep 11 22:21:26.494822 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.494801    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 22:21:26.500417 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.500397    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"]
Sep 11 22:21:26.520019 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.519989    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:26.621294 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.621256    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:26.624577 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.624556    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:26.624713 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.624661    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1ddc21-2a91-4165-b942-cf751f05852e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:26.624713 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.624685    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-metrics-certs\") pod \"maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:26.624830 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.624806    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bfdp\" (UniqueName: \"kubernetes.io/projected/fb1ddc21-2a91-4165-b942-cf751f05852e-kube-api-access-5bfdp\") pod \"maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:26.725885 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.725860    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1ddc21-2a91-4165-b942-cf751f05852e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:26.726232 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.725892    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-metrics-certs\") pod \"maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:26.726232 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.725936    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bfdp\" (UniqueName: \"kubernetes.io/projected/fb1ddc21-2a91-4165-b942-cf751f05852e-kube-api-access-5bfdp\") pod \"maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:26.726232 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.725966    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:26.726232 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:21:26.726081    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-serving-cert: secret "maas-api-e2e-ipp-a-6fe4c2-serving-cert" not found
Sep 11 22:21:26.726232 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:21:26.726151    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-maas-api-tls podName:fb1ddc21-2a91-4165-b942-cf751f05852e nodeName:}" failed. No retries permitted until 2026-09-11 22:21:27.226132829 +0000 UTC m=+1753.961244152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-maas-api-tls") pod "maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47" (UID: "fb1ddc21-2a91-4165-b942-cf751f05852e") : secret "maas-api-e2e-ipp-a-6fe4c2-serving-cert" not found
Sep 11 22:21:26.726525 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.726500    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1ddc21-2a91-4165-b942-cf751f05852e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:26.728330 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.728296    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-metrics-certs\") pod \"maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:26.734663 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.734643    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bfdp\" (UniqueName: \"kubernetes.io/projected/fb1ddc21-2a91-4165-b942-cf751f05852e-kube-api-access-5bfdp\") pod \"maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:26.975914 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:26.975883    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:27.076855 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.076819    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:27.108321 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.108274    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"]
Sep 11 22:21:27.112045 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.112026    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"
Sep 11 22:21:27.115533 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.115515    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6fe4c2-dockercfg-qh6f2\""
Sep 11 22:21:27.115700 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.115555    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6fe4c2\""
Sep 11 22:21:27.123353 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.123326    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"]
Sep 11 22:21:27.126393 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.126368    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"]
Sep 11 22:21:27.129954 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.129936    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"
Sep 11 22:21:27.140613 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.140588    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"]
Sep 11 22:21:27.169098 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.169077    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:27.230499 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.230424    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b6g2\" (UniqueName: \"kubernetes.io/projected/5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7-kube-api-access-4b6g2\") pod \"payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq\" (UID: \"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"
Sep 11 22:21:27.230499 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.230460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd986702-2d3c-4c48-b2f6-7b01417357d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl\" (UID: \"cd986702-2d3c-4c48-b2f6-7b01417357d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"
Sep 11 22:21:27.230679 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.230501    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq\" (UID: \"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"
Sep 11 22:21:27.230679 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.230543    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:27.230679 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.230627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4mzz\" (UniqueName: \"kubernetes.io/projected/cd986702-2d3c-4c48-b2f6-7b01417357d8-kube-api-access-k4mzz\") pod \"payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl\" (UID: \"cd986702-2d3c-4c48-b2f6-7b01417357d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"
Sep 11 22:21:27.232885 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.232862    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:27.331921 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.331890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b6g2\" (UniqueName: \"kubernetes.io/projected/5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7-kube-api-access-4b6g2\") pod \"payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq\" (UID: \"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"
Sep 11 22:21:27.332087 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.331926    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd986702-2d3c-4c48-b2f6-7b01417357d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl\" (UID: \"cd986702-2d3c-4c48-b2f6-7b01417357d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"
Sep 11 22:21:27.332087 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.331952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq\" (UID: \"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"
Sep 11 22:21:27.332087 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.332025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4mzz\" (UniqueName: \"kubernetes.io/projected/cd986702-2d3c-4c48-b2f6-7b01417357d8-kube-api-access-k4mzz\") pod \"payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl\" (UID: \"cd986702-2d3c-4c48-b2f6-7b01417357d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"
Sep 11 22:21:27.332676 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.332651    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd986702-2d3c-4c48-b2f6-7b01417357d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl\" (UID: \"cd986702-2d3c-4c48-b2f6-7b01417357d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"
Sep 11 22:21:27.332755 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.332738    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq\" (UID: \"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"
Sep 11 22:21:27.340950 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.340916    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4mzz\" (UniqueName: \"kubernetes.io/projected/cd986702-2d3c-4c48-b2f6-7b01417357d8-kube-api-access-k4mzz\") pod \"payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl\" (UID: \"cd986702-2d3c-4c48-b2f6-7b01417357d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"
Sep 11 22:21:27.341093 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.341072    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b6g2\" (UniqueName: \"kubernetes.io/projected/5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7-kube-api-access-4b6g2\") pod \"payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq\" (UID: \"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"
Sep 11 22:21:27.402285 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.402259    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:27.425393 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.425362    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"
Sep 11 22:21:27.439583 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.439559    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"
Sep 11 22:21:27.679502 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.679470    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"]
Sep 11 22:21:27.682388 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:21:27.682361    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1ddc21_2a91_4165_b942_cf751f05852e.slice/crio-091c1af288ab3c3753c15073767c66ba65f452b23e9312caf01c165b5a1258c1 WatchSource:0}: Error finding container 091c1af288ab3c3753c15073767c66ba65f452b23e9312caf01c165b5a1258c1: Status 404 returned error can't find the container with id 091c1af288ab3c3753c15073767c66ba65f452b23e9312caf01c165b5a1258c1
Sep 11 22:21:27.709465 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.709441    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"]
Sep 11 22:21:27.712417 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:21:27.712383    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5d04e5_f8b1_4d9a_b89f_15a58ead08e7.slice/crio-425c9fa38e9d7e1080d597b716c16c58dd36d1220c04c81636e8f692251c477d WatchSource:0}: Error finding container 425c9fa38e9d7e1080d597b716c16c58dd36d1220c04c81636e8f692251c477d: Status 404 returned error can't find the container with id 425c9fa38e9d7e1080d597b716c16c58dd36d1220c04c81636e8f692251c477d
Sep 11 22:21:27.728574 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:27.728551    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"]
Sep 11 22:21:27.730616 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:21:27.730591    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd986702_2d3c_4c48_b2f6_7b01417357d8.slice/crio-c361fc313f167a8ca64591a09f6c3a3fb9287a3f1c5f098897a60b7b49a26aab WatchSource:0}: Error finding container c361fc313f167a8ca64591a09f6c3a3fb9287a3f1c5f098897a60b7b49a26aab: Status 404 returned error can't find the container with id c361fc313f167a8ca64591a09f6c3a3fb9287a3f1c5f098897a60b7b49a26aab
Sep 11 22:21:28.191992 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:28.191956    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47" event={"ID":"fb1ddc21-2a91-4165-b942-cf751f05852e","Type":"ContainerStarted","Data":"091c1af288ab3c3753c15073767c66ba65f452b23e9312caf01c165b5a1258c1"}
Sep 11 22:21:28.193889 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:28.193834    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl" event={"ID":"cd986702-2d3c-4c48-b2f6-7b01417357d8","Type":"ContainerStarted","Data":"4f85788e3724326e812e1f4abc5ef96a8d37b161e4d9b775af45c08af043d7e9"}
Sep 11 22:21:28.193889 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:28.193880    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl" event={"ID":"cd986702-2d3c-4c48-b2f6-7b01417357d8","Type":"ContainerStarted","Data":"c361fc313f167a8ca64591a09f6c3a3fb9287a3f1c5f098897a60b7b49a26aab"}
Sep 11 22:21:28.194069 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:28.194054    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"
Sep 11 22:21:28.195666 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:28.195645    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq" event={"ID":"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7","Type":"ContainerStarted","Data":"c3dc53d6174dad2c3751067e68d2f4fd2463a8f69d8590a2105e73247a802d03"}
Sep 11 22:21:28.195763 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:28.195695    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq" event={"ID":"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7","Type":"ContainerStarted","Data":"425c9fa38e9d7e1080d597b716c16c58dd36d1220c04c81636e8f692251c477d"}
Sep 11 22:21:28.195842 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:28.195810    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"
Sep 11 22:21:28.211882 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:28.211833    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl" podStartSLOduration=1.211817559 podStartE2EDuration="1.211817559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:21:28.211177032 +0000 UTC m=+1754.946288380" watchObservedRunningTime="2026-09-11 22:21:28.211817559 +0000 UTC m=+1754.946928906"
Sep 11 22:21:28.235491 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:28.235448    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq" podStartSLOduration=1.235433645 podStartE2EDuration="1.235433645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:21:28.233756022 +0000 UTC m=+1754.968867372" watchObservedRunningTime="2026-09-11 22:21:28.235433645 +0000 UTC m=+1754.970544993"
Sep 11 22:21:29.201778 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:29.201742    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47" event={"ID":"fb1ddc21-2a91-4165-b942-cf751f05852e","Type":"ContainerStarted","Data":"5408e8ffd7056c68379c37cc8ef481ce6f0e86a122cd13d734dd8f9c433c2836"}
Sep 11 22:21:29.202192 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:29.201800    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:29.233196 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:29.233142    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47" podStartSLOduration=2.789855545 podStartE2EDuration="3.233121836s" totalImagesPullingTime="443.266291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:21:29.229937918 +0000 UTC m=+1755.965049265" watchObservedRunningTime="2026-09-11 22:21:29.233121836 +0000 UTC m=+1755.968233188"
Sep 11 22:21:35.212591 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:35.212557    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:21:35.885328 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:35.885280    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52" podUID="e862dd9d-962c-4b90-926f-04723bac28a7" containerName="main" containerID="cri-o://ff5bbbcf8d15981ca620ac37cb54bea028e00c41b1b1c532589d3084e67ae64e" gracePeriod=45
Sep 11 22:21:36.137645 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.137591    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:21:36.239103 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.239074    2574 generic.go:356] "Generic (PLEG): container finished" podID="e862dd9d-962c-4b90-926f-04723bac28a7" containerID="ff5bbbcf8d15981ca620ac37cb54bea028e00c41b1b1c532589d3084e67ae64e" exitCode=0
Sep 11 22:21:36.239477 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.239109    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"
Sep 11 22:21:36.239477 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.239156    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52" event={"ID":"e862dd9d-962c-4b90-926f-04723bac28a7","Type":"ContainerDied","Data":"ff5bbbcf8d15981ca620ac37cb54bea028e00c41b1b1c532589d3084e67ae64e"}
Sep 11 22:21:36.239477 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.239196    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52" event={"ID":"e862dd9d-962c-4b90-926f-04723bac28a7","Type":"ContainerDied","Data":"7dd38f23359b322a6494c30d81933907cb9e72b1abc769b0a7d8c1923331b13f"}
Sep 11 22:21:36.239477 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.239214    2574 scope.go:117] "RemoveContainer" containerID="ff5bbbcf8d15981ca620ac37cb54bea028e00c41b1b1c532589d3084e67ae64e"
Sep 11 22:21:36.247507 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.247488    2574 scope.go:117] "RemoveContainer" containerID="ff5bbbcf8d15981ca620ac37cb54bea028e00c41b1b1c532589d3084e67ae64e"
Sep 11 22:21:36.247755 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:21:36.247739    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5bbbcf8d15981ca620ac37cb54bea028e00c41b1b1c532589d3084e67ae64e\": container with ID starting with ff5bbbcf8d15981ca620ac37cb54bea028e00c41b1b1c532589d3084e67ae64e not found: ID does not exist" containerID="ff5bbbcf8d15981ca620ac37cb54bea028e00c41b1b1c532589d3084e67ae64e"
Sep 11 22:21:36.247814 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.247765    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5bbbcf8d15981ca620ac37cb54bea028e00c41b1b1c532589d3084e67ae64e"} err="failed to get container status \"ff5bbbcf8d15981ca620ac37cb54bea028e00c41b1b1c532589d3084e67ae64e\": rpc error: code = NotFound desc = could not find container \"ff5bbbcf8d15981ca620ac37cb54bea028e00c41b1b1c532589d3084e67ae64e\": container with ID starting with ff5bbbcf8d15981ca620ac37cb54bea028e00c41b1b1c532589d3084e67ae64e not found: ID does not exist"
Sep 11 22:21:36.316139 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.316113    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-tmp-dir\") pod \"e862dd9d-962c-4b90-926f-04723bac28a7\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") "
Sep 11 22:21:36.316254 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.316146    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-model-cache\") pod \"e862dd9d-962c-4b90-926f-04723bac28a7\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") "
Sep 11 22:21:36.316254 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.316204    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-dshm\") pod \"e862dd9d-962c-4b90-926f-04723bac28a7\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") "
Sep 11 22:21:36.316254 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.316226    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e862dd9d-962c-4b90-926f-04723bac28a7-tls-certs\") pod \"e862dd9d-962c-4b90-926f-04723bac28a7\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") "
Sep 11 22:21:36.316441 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.316267    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-home\") pod \"e862dd9d-962c-4b90-926f-04723bac28a7\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") "
Sep 11 22:21:36.316441 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.316335    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw4k2\" (UniqueName: \"kubernetes.io/projected/e862dd9d-962c-4b90-926f-04723bac28a7-kube-api-access-kw4k2\") pod \"e862dd9d-962c-4b90-926f-04723bac28a7\" (UID: \"e862dd9d-962c-4b90-926f-04723bac28a7\") "
Sep 11 22:21:36.316564 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.316448    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e862dd9d-962c-4b90-926f-04723bac28a7" (UID: "e862dd9d-962c-4b90-926f-04723bac28a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:21:36.316564 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.316474    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e862dd9d-962c-4b90-926f-04723bac28a7" (UID: "e862dd9d-962c-4b90-926f-04723bac28a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:21:36.316652 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.316593    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-home" (OuterVolumeSpecName: "home") pod "e862dd9d-962c-4b90-926f-04723bac28a7" (UID: "e862dd9d-962c-4b90-926f-04723bac28a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:21:36.316730 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.316708    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-home\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:21:36.316788 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.316729    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-tmp-dir\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:21:36.316788 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.316740    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-model-cache\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:21:36.318511 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.318487    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e862dd9d-962c-4b90-926f-04723bac28a7-kube-api-access-kw4k2" (OuterVolumeSpecName: "kube-api-access-kw4k2") pod "e862dd9d-962c-4b90-926f-04723bac28a7" (UID: "e862dd9d-962c-4b90-926f-04723bac28a7"). InnerVolumeSpecName "kube-api-access-kw4k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:21:36.318881 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.318861    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-dshm" (OuterVolumeSpecName: "dshm") pod "e862dd9d-962c-4b90-926f-04723bac28a7" (UID: "e862dd9d-962c-4b90-926f-04723bac28a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:21:36.318958 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.318857    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e862dd9d-962c-4b90-926f-04723bac28a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e862dd9d-962c-4b90-926f-04723bac28a7" (UID: "e862dd9d-962c-4b90-926f-04723bac28a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:21:36.417558 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.417501    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e862dd9d-962c-4b90-926f-04723bac28a7-tls-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:21:36.417558 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.417523    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw4k2\" (UniqueName: \"kubernetes.io/projected/e862dd9d-962c-4b90-926f-04723bac28a7-kube-api-access-kw4k2\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:21:36.417558 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.417531    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e862dd9d-962c-4b90-926f-04723bac28a7-dshm\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:21:36.560735 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.560705    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"]
Sep 11 22:21:36.564690 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:36.564670    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/rate-test-model-966527-kserve-66df9846c9-6ds52"]
Sep 11 22:21:37.720743 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:37.720702    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e862dd9d-962c-4b90-926f-04723bac28a7" path="/var/lib/kubelet/pods/e862dd9d-962c-4b90-926f-04723bac28a7/volumes"
Sep 11 22:21:39.204132 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:39.204099    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"
Sep 11 22:21:39.204484 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:39.204176    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"
Sep 11 22:21:41.733031 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:41.733002    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:47.361213 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:47.361169    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:49.004951 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:49.004923    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:49.705681 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:49.705634    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:49.921115 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:49.921057    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:50.435054 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:50.434989    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:50.588028 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:50.587997    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:50.894533 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:50.894495    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:51.217136 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:51.217041    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:51.405668 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:51.405634    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:53.817050 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:53.816992    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:54.033840 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:54.033799    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:54.131488 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:54.131452    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:54.762363 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:54.762323    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:55.513679 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:55.513638    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:56.262195 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.262160    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"]
Sep 11 22:21:56.262648 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.262633    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e862dd9d-962c-4b90-926f-04723bac28a7" containerName="main"
Sep 11 22:21:56.262648 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.262649    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e862dd9d-962c-4b90-926f-04723bac28a7" containerName="main"
Sep 11 22:21:56.262765 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.262758    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e862dd9d-962c-4b90-926f-04723bac28a7" containerName="main"
Sep 11 22:21:56.265987 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.265969    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:56.268955 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.268929    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-dbaf52-serving-cert\""
Sep 11 22:21:56.279000 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.278955    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"]
Sep 11 22:21:56.295558 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.295536    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:56.391014 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.390982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmnnz\" (UniqueName: \"kubernetes.io/projected/7969118d-e553-4314-aacc-99dc73f011ac-kube-api-access-bmnnz\") pod \"maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:56.391014 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.391013    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7969118d-e553-4314-aacc-99dc73f011ac-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:56.391211 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.391038    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:56.391211 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.391068    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-metrics-certs\") pod \"maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:56.395083 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.395061    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:56.491885 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.491863    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmnnz\" (UniqueName: \"kubernetes.io/projected/7969118d-e553-4314-aacc-99dc73f011ac-kube-api-access-bmnnz\") pod \"maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:56.491998 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.491893    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7969118d-e553-4314-aacc-99dc73f011ac-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:56.491998 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.491919    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:56.491998 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.491959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-metrics-certs\") pod \"maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:56.492131 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:21:56.492039    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-serving-cert: secret "maas-api-e2e-ipp-b-dbaf52-serving-cert" not found
Sep 11 22:21:56.492131 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:21:56.492098    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-maas-api-tls podName:7969118d-e553-4314-aacc-99dc73f011ac nodeName:}" failed. No retries permitted until 2026-09-11 22:21:56.99208215 +0000 UTC m=+1783.727193474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-maas-api-tls") pod "maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5" (UID: "7969118d-e553-4314-aacc-99dc73f011ac") : secret "maas-api-e2e-ipp-b-dbaf52-serving-cert" not found
Sep 11 22:21:56.492646 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.492623    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7969118d-e553-4314-aacc-99dc73f011ac-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:56.494379 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.494359    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-metrics-certs\") pod \"maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:56.502570 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.502551    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmnnz\" (UniqueName: \"kubernetes.io/projected/7969118d-e553-4314-aacc-99dc73f011ac-kube-api-access-bmnnz\") pod \"maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:56.778286 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.778251    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:56.878571 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.878537    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:21:56.916441 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.916409    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"]
Sep 11 22:21:56.920724 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.920699    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"
Sep 11 22:21:56.924176 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.924141    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dbaf52-dockercfg-7vrf6\""
Sep 11 22:21:56.924342 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.924215    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dbaf52\""
Sep 11 22:21:56.932405 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.932382    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"]
Sep 11 22:21:56.996124 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.996093    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:56.996258 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.996137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17c845ce-c7e7-4d72-930c-d7200d14175b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr\" (UID: \"17c845ce-c7e7-4d72-930c-d7200d14175b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"
Sep 11 22:21:56.996258 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.996188    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjwhx\" (UniqueName: \"kubernetes.io/projected/17c845ce-c7e7-4d72-930c-d7200d14175b-kube-api-access-tjwhx\") pod \"payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr\" (UID: \"17c845ce-c7e7-4d72-930c-d7200d14175b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"
Sep 11 22:21:56.998452 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:56.998427    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:57.096878 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:57.096845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17c845ce-c7e7-4d72-930c-d7200d14175b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr\" (UID: \"17c845ce-c7e7-4d72-930c-d7200d14175b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"
Sep 11 22:21:57.097047 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:57.096945    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjwhx\" (UniqueName: \"kubernetes.io/projected/17c845ce-c7e7-4d72-930c-d7200d14175b-kube-api-access-tjwhx\") pod \"payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr\" (UID: \"17c845ce-c7e7-4d72-930c-d7200d14175b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"
Sep 11 22:21:57.097622 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:57.097603    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17c845ce-c7e7-4d72-930c-d7200d14175b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr\" (UID: \"17c845ce-c7e7-4d72-930c-d7200d14175b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"
Sep 11 22:21:57.106208 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:57.106182    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjwhx\" (UniqueName: \"kubernetes.io/projected/17c845ce-c7e7-4d72-930c-d7200d14175b-kube-api-access-tjwhx\") pod \"payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr\" (UID: \"17c845ce-c7e7-4d72-930c-d7200d14175b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"
Sep 11 22:21:57.177001 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:57.176962    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:57.233506 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:57.233478    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"
Sep 11 22:21:57.451276 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:57.451251    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"]
Sep 11 22:21:57.453029 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:21:57.453007    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7969118d_e553_4314_aacc_99dc73f011ac.slice/crio-2022715372e3858ed7f68a34f39da0bdb16f97e507483d60b7cd79f1e75f541f WatchSource:0}: Error finding container 2022715372e3858ed7f68a34f39da0bdb16f97e507483d60b7cd79f1e75f541f: Status 404 returned error can't find the container with id 2022715372e3858ed7f68a34f39da0bdb16f97e507483d60b7cd79f1e75f541f
Sep 11 22:21:57.454392 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:57.454375    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:21:57.508359 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:57.508321    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"]
Sep 11 22:21:57.510287 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:21:57.510262    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c845ce_c7e7_4d72_930c_d7200d14175b.slice/crio-e407e988b8d263c9ce60c95a06371fe71a8a535f303ea8257f4f40feeda1326a WatchSource:0}: Error finding container e407e988b8d263c9ce60c95a06371fe71a8a535f303ea8257f4f40feeda1326a: Status 404 returned error can't find the container with id e407e988b8d263c9ce60c95a06371fe71a8a535f303ea8257f4f40feeda1326a
Sep 11 22:21:58.357322 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:58.357266    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr" event={"ID":"17c845ce-c7e7-4d72-930c-d7200d14175b","Type":"ContainerStarted","Data":"ec467a8cf547ee0f78aa1170c5e6e14657a81b07d233d83ed57c4c59ade4e5ad"}
Sep 11 22:21:58.357802 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:58.357359    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"
Sep 11 22:21:58.357802 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:58.357383    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr" event={"ID":"17c845ce-c7e7-4d72-930c-d7200d14175b","Type":"ContainerStarted","Data":"e407e988b8d263c9ce60c95a06371fe71a8a535f303ea8257f4f40feeda1326a"}
Sep 11 22:21:58.359303 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:58.359278    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5" event={"ID":"7969118d-e553-4314-aacc-99dc73f011ac","Type":"ContainerStarted","Data":"bb21acf189550947864276758902574283c6dee7799563766627c54089f7bf5c"}
Sep 11 22:21:58.359458 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:58.359350    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5" event={"ID":"7969118d-e553-4314-aacc-99dc73f011ac","Type":"ContainerStarted","Data":"2022715372e3858ed7f68a34f39da0bdb16f97e507483d60b7cd79f1e75f541f"}
Sep 11 22:21:58.359458 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:58.359387    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:21:58.383300 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:58.383254    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr" podStartSLOduration=2.3832375040000002 podStartE2EDuration="2.383237504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:21:58.382168672 +0000 UTC m=+1785.117280019" watchObservedRunningTime="2026-09-11 22:21:58.383237504 +0000 UTC m=+1785.118348853"
Sep 11 22:21:58.404521 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:21:58.404476    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5" podStartSLOduration=2.022206595 podStartE2EDuration="2.404460378s" totalImagesPullingTime="382.253783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:21:58.401989615 +0000 UTC m=+1785.137100960" watchObservedRunningTime="2026-09-11 22:21:58.404460378 +0000 UTC m=+1785.139571727"
Sep 11 22:22:04.370527 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:04.370498    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:22:09.367531 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:09.367444    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"
Sep 11 22:22:13.812610 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:13.812582    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:22:13.816806 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:13.816783    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:22:13.817800 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:13.817782    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:22:13.822649 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:13.822634    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:22:14.279278 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:14.279236    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:22:17.203940 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:17.203904    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:22:18.713480 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:18.713446    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:22:19.058627 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:19.058541    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:22:19.627161 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:19.627122    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:22:20.301020 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:20.300966    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:22:21.064445 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:21.064380    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:22:21.811518 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:21.811480    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:22:24.214728 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:24.214699    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr_17c845ce-c7e7-4d72-930c-d7200d14175b/payload-processing/0.log"
Sep 11 22:22:27.704691 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:27.704641    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:22:50.404631 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:22:50.404592    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:00.102825 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:00.102796    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl_cd986702-2d3c-4c48-b2f6-7b01417357d8/payload-processing/0.log"
Sep 11 22:23:00.329743 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:00.329716    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr_17c845ce-c7e7-4d72-930c-d7200d14175b/payload-processing/0.log"
Sep 11 22:23:00.614819 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:00.614782    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:01.736092 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:01.736051    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:03.889498 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:03.889460    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:04.466418 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:04.466369    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:04.565699 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:04.565670    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:04.723616 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:04.723529    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:05.428969 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:05.428931    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:06.303644 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:06.303615    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:07.295223 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:07.295185    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:07.884148 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:07.884117    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"]
Sep 11 22:23:07.888095 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:07.888069    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:07.890724 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:07.890702    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-1688a8-serving-cert\""
Sep 11 22:23:07.895625 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:07.895602    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"]
Sep 11 22:23:07.913102 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:07.913077    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:07.970837 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:07.970810    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:08.036699 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.036670    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:08.036856 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.036713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:08.036856 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.036750    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blcff\" (UniqueName: \"kubernetes.io/projected/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-kube-api-access-blcff\") pod \"maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:08.036931 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.036860    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:08.137678 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.137607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:08.137678 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.137655    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:08.137909 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.137690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:08.137909 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.137725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blcff\" (UniqueName: \"kubernetes.io/projected/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-kube-api-access-blcff\") pod \"maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:08.138420 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.138393    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:08.139962 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.139942    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:08.140052 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.139970    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:08.145328 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.145284    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blcff\" (UniqueName: \"kubernetes.io/projected/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-kube-api-access-blcff\") pod \"maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:08.152283 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.152255    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:08.201241 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.201217    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:08.361636 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.361601    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:08.462389 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.462249    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:08.474145 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.474118    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"]
Sep 11 22:23:08.475671 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:23:08.475648    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f3ecf9_88fb_48eb_b30d_fa9c62dd4141.slice/crio-9545571c734bfab9cceb82371dd614622808469399d740e2020044748cde2443 WatchSource:0}: Error finding container 9545571c734bfab9cceb82371dd614622808469399d740e2020044748cde2443: Status 404 returned error can't find the container with id 9545571c734bfab9cceb82371dd614622808469399d740e2020044748cde2443
Sep 11 22:23:08.718831 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:08.718747    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5" event={"ID":"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141","Type":"ContainerStarted","Data":"9545571c734bfab9cceb82371dd614622808469399d740e2020044748cde2443"}
Sep 11 22:23:09.633574 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:09.633537    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:09.725461 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:09.725417    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5" event={"ID":"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141","Type":"ContainerStarted","Data":"080e2dbd5b5d2d79a8420e188dbdd76cf4c41c3ad4d074b4a8a1bf39b9b3e428"}
Sep 11 22:23:09.726071 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:09.726045    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:09.746635 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:09.746590    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5" podStartSLOduration=2.26374836 podStartE2EDuration="2.746577322s" totalImagesPullingTime="482.828962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:23:09.743398938 +0000 UTC m=+1856.478510283" watchObservedRunningTime="2026-09-11 22:23:09.746577322 +0000 UTC m=+1856.481688668"
Sep 11 22:23:15.736985 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:15.736958    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:30.035422 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:30.035388    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:30.801836 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:30.801794    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:31.720392 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:31.720358    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:32.395852 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:32.395815    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:33.043915 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:33.043880    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:33.819461 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:33.819426    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:34.554092 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:34.554060    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:35.973514 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:35.973479    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:36.705751 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:36.705706    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:37.336668 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:37.336607    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:38.085969 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:38.085930    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:38.817166 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:38.817119    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:39.462757 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:39.462719    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:46.120340 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.120294    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"]
Sep 11 22:23:46.121007 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.120521    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5" podUID="56f3ecf9-88fb-48eb-b30d-fa9c62dd4141" containerName="maas-api" containerID="cri-o://080e2dbd5b5d2d79a8420e188dbdd76cf4c41c3ad4d074b4a8a1bf39b9b3e428" gracePeriod=30
Sep 11 22:23:46.153144 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.153113    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:46.220020 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.219960    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:46.282531 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.282426    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:46.338861 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.338823    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:46.373865 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.373816    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:46.406444 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.406419    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:46.469798 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.469772    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-maas-api-tls\") pod \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") "
Sep 11 22:23:46.470045 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.469815    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blcff\" (UniqueName: \"kubernetes.io/projected/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-kube-api-access-blcff\") pod \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") "
Sep 11 22:23:46.470045 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.469871    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-openshift-service-ca\") pod \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") "
Sep 11 22:23:46.470045 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.469936    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-metrics-certs\") pod \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\" (UID: \"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141\") "
Sep 11 22:23:46.470330 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.470291    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56f3ecf9-88fb-48eb-b30d-fa9c62dd4141" (UID: "56f3ecf9-88fb-48eb-b30d-fa9c62dd4141"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:23:46.470610 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.470592    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-openshift-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:23:46.472196 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.472166    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "56f3ecf9-88fb-48eb-b30d-fa9c62dd4141" (UID: "56f3ecf9-88fb-48eb-b30d-fa9c62dd4141"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:23:46.472537 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.472516    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-kube-api-access-blcff" (OuterVolumeSpecName: "kube-api-access-blcff") pod "56f3ecf9-88fb-48eb-b30d-fa9c62dd4141" (UID: "56f3ecf9-88fb-48eb-b30d-fa9c62dd4141"). InnerVolumeSpecName "kube-api-access-blcff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:23:46.472614 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.472539    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56f3ecf9-88fb-48eb-b30d-fa9c62dd4141" (UID: "56f3ecf9-88fb-48eb-b30d-fa9c62dd4141"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:23:46.571736 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.571702    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-maas-api-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:23:46.571736 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.571729    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blcff\" (UniqueName: \"kubernetes.io/projected/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-kube-api-access-blcff\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:23:46.571736 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.571739    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141-metrics-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:23:46.715293 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.715216    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:46.821114 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.821076    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:46.906905 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.906867    2574 generic.go:356] "Generic (PLEG): container finished" podID="56f3ecf9-88fb-48eb-b30d-fa9c62dd4141" containerID="080e2dbd5b5d2d79a8420e188dbdd76cf4c41c3ad4d074b4a8a1bf39b9b3e428" exitCode=0
Sep 11 22:23:46.907079 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.906921    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"
Sep 11 22:23:46.907079 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.906948    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5" event={"ID":"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141","Type":"ContainerDied","Data":"080e2dbd5b5d2d79a8420e188dbdd76cf4c41c3ad4d074b4a8a1bf39b9b3e428"}
Sep 11 22:23:46.907079 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.907010    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5" event={"ID":"56f3ecf9-88fb-48eb-b30d-fa9c62dd4141","Type":"ContainerDied","Data":"9545571c734bfab9cceb82371dd614622808469399d740e2020044748cde2443"}
Sep 11 22:23:46.907079 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.907041    2574 scope.go:117] "RemoveContainer" containerID="080e2dbd5b5d2d79a8420e188dbdd76cf4c41c3ad4d074b4a8a1bf39b9b3e428"
Sep 11 22:23:46.917165 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.917146    2574 scope.go:117] "RemoveContainer" containerID="080e2dbd5b5d2d79a8420e188dbdd76cf4c41c3ad4d074b4a8a1bf39b9b3e428"
Sep 11 22:23:46.917473 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:23:46.917455    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"080e2dbd5b5d2d79a8420e188dbdd76cf4c41c3ad4d074b4a8a1bf39b9b3e428\": container with ID starting with 080e2dbd5b5d2d79a8420e188dbdd76cf4c41c3ad4d074b4a8a1bf39b9b3e428 not found: ID does not exist" containerID="080e2dbd5b5d2d79a8420e188dbdd76cf4c41c3ad4d074b4a8a1bf39b9b3e428"
Sep 11 22:23:46.917546 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.917483    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080e2dbd5b5d2d79a8420e188dbdd76cf4c41c3ad4d074b4a8a1bf39b9b3e428"} err="failed to get container status \"080e2dbd5b5d2d79a8420e188dbdd76cf4c41c3ad4d074b4a8a1bf39b9b3e428\": rpc error: code = NotFound desc = could not find container \"080e2dbd5b5d2d79a8420e188dbdd76cf4c41c3ad4d074b4a8a1bf39b9b3e428\": container with ID starting with 080e2dbd5b5d2d79a8420e188dbdd76cf4c41c3ad4d074b4a8a1bf39b9b3e428 not found: ID does not exist"
Sep 11 22:23:46.930198 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.930176    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"]
Sep 11 22:23:46.933798 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:46.933776    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1688a8-b9f4bbf6f-f6xv5"]
Sep 11 22:23:47.587911 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:47.587871    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:47.722038 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:47.722010    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56f3ecf9-88fb-48eb-b30d-fa9c62dd4141" path="/var/lib/kubelet/pods/56f3ecf9-88fb-48eb-b30d-fa9c62dd4141/volumes"
Sep 11 22:23:48.300344 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:48.300273    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:48.998142 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:48.998097    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:49.728063 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:49.728031    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:52.380428 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:52.380392    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:52.441825 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:52.441796    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:52.640260 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:52.640184    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:56.593888 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:56.593848    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:57.508785 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:57.508750    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:58.202136 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:58.202093    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:58.912895 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:58.912859    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:23:59.658532 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:23:59.658494    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:00.341053 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:00.341021    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:11.846351 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:11.846288    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"]
Sep 11 22:24:11.846823 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:11.846707    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47" podUID="fb1ddc21-2a91-4165-b942-cf751f05852e" containerName="maas-api" containerID="cri-o://5408e8ffd7056c68379c37cc8ef481ce6f0e86a122cd13d734dd8f9c433c2836" gracePeriod=30
Sep 11 22:24:11.878320 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:11.878278    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:11.926302 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:11.926269    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"]
Sep 11 22:24:11.926533 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:11.926508    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl" podUID="cd986702-2d3c-4c48-b2f6-7b01417357d8" containerName="payload-processing" containerID="cri-o://4f85788e3724326e812e1f4abc5ef96a8d37b161e4d9b775af45c08af043d7e9" gracePeriod=30
Sep 11 22:24:11.943602 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:11.943568    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"]
Sep 11 22:24:11.943932 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:11.943818    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq" podUID="5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7" containerName="payload-pre-processing" containerID="cri-o://c3dc53d6174dad2c3751067e68d2f4fd2463a8f69d8590a2105e73247a802d03" gracePeriod=30
Sep 11 22:24:11.989800 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:11.989763    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:12.043002 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.042970    2574 generic.go:356] "Generic (PLEG): container finished" podID="fb1ddc21-2a91-4165-b942-cf751f05852e" containerID="5408e8ffd7056c68379c37cc8ef481ce6f0e86a122cd13d734dd8f9c433c2836" exitCode=0
Sep 11 22:24:12.043125 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.043049    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47" event={"ID":"fb1ddc21-2a91-4165-b942-cf751f05852e","Type":"ContainerDied","Data":"5408e8ffd7056c68379c37cc8ef481ce6f0e86a122cd13d734dd8f9c433c2836"}
Sep 11 22:24:12.044955 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.044933    2574 generic.go:356] "Generic (PLEG): container finished" podID="cd986702-2d3c-4c48-b2f6-7b01417357d8" containerID="4f85788e3724326e812e1f4abc5ef96a8d37b161e4d9b775af45c08af043d7e9" exitCode=0
Sep 11 22:24:12.045042 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.044990    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl" event={"ID":"cd986702-2d3c-4c48-b2f6-7b01417357d8","Type":"ContainerDied","Data":"4f85788e3724326e812e1f4abc5ef96a8d37b161e4d9b775af45c08af043d7e9"}
Sep 11 22:24:12.093779 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.093740    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:12.161446 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.161199    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:12.209256 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.209235    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:24:12.212430 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.212405    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"
Sep 11 22:24:12.224577 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.224500    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:12.246765 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.246741    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"
Sep 11 22:24:12.306094 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.306063    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4mzz\" (UniqueName: \"kubernetes.io/projected/cd986702-2d3c-4c48-b2f6-7b01417357d8-kube-api-access-k4mzz\") pod \"cd986702-2d3c-4c48-b2f6-7b01417357d8\" (UID: \"cd986702-2d3c-4c48-b2f6-7b01417357d8\") "
Sep 11 22:24:12.306287 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.306129    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1ddc21-2a91-4165-b942-cf751f05852e-openshift-service-ca\") pod \"fb1ddc21-2a91-4165-b942-cf751f05852e\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") "
Sep 11 22:24:12.306287 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.306166    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-maas-api-tls\") pod \"fb1ddc21-2a91-4165-b942-cf751f05852e\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") "
Sep 11 22:24:12.306287 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.306187    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b6g2\" (UniqueName: \"kubernetes.io/projected/5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7-kube-api-access-4b6g2\") pod \"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7\" (UID: \"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7\") "
Sep 11 22:24:12.306287 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.306208    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-metrics-certs\") pod \"fb1ddc21-2a91-4165-b942-cf751f05852e\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") "
Sep 11 22:24:12.306287 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.306256    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd986702-2d3c-4c48-b2f6-7b01417357d8-plugins-config-volume\") pod \"cd986702-2d3c-4c48-b2f6-7b01417357d8\" (UID: \"cd986702-2d3c-4c48-b2f6-7b01417357d8\") "
Sep 11 22:24:12.306287 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.306280    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bfdp\" (UniqueName: \"kubernetes.io/projected/fb1ddc21-2a91-4165-b942-cf751f05852e-kube-api-access-5bfdp\") pod \"fb1ddc21-2a91-4165-b942-cf751f05852e\" (UID: \"fb1ddc21-2a91-4165-b942-cf751f05852e\") "
Sep 11 22:24:12.306623 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.306389    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7-plugins-config-volume\") pod \"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7\" (UID: \"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7\") "
Sep 11 22:24:12.306623 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.306569    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1ddc21-2a91-4165-b942-cf751f05852e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb1ddc21-2a91-4165-b942-cf751f05852e" (UID: "fb1ddc21-2a91-4165-b942-cf751f05852e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:24:12.306930 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.306902    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7" (UID: "5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:24:12.307037 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.306913    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1ddc21-2a91-4165-b942-cf751f05852e-openshift-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:24:12.307094 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.307027    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd986702-2d3c-4c48-b2f6-7b01417357d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cd986702-2d3c-4c48-b2f6-7b01417357d8" (UID: "cd986702-2d3c-4c48-b2f6-7b01417357d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:24:12.308653 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.308628    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd986702-2d3c-4c48-b2f6-7b01417357d8-kube-api-access-k4mzz" (OuterVolumeSpecName: "kube-api-access-k4mzz") pod "cd986702-2d3c-4c48-b2f6-7b01417357d8" (UID: "cd986702-2d3c-4c48-b2f6-7b01417357d8"). InnerVolumeSpecName "kube-api-access-k4mzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:24:12.308796 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.308779    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fb1ddc21-2a91-4165-b942-cf751f05852e" (UID: "fb1ddc21-2a91-4165-b942-cf751f05852e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:24:12.308912 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.308895    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7-kube-api-access-4b6g2" (OuterVolumeSpecName: "kube-api-access-4b6g2") pod "5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7" (UID: "5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7"). InnerVolumeSpecName "kube-api-access-4b6g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:24:12.308969 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.308911    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1ddc21-2a91-4165-b942-cf751f05852e-kube-api-access-5bfdp" (OuterVolumeSpecName: "kube-api-access-5bfdp") pod "fb1ddc21-2a91-4165-b942-cf751f05852e" (UID: "fb1ddc21-2a91-4165-b942-cf751f05852e"). InnerVolumeSpecName "kube-api-access-5bfdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:24:12.309052 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.309038    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb1ddc21-2a91-4165-b942-cf751f05852e" (UID: "fb1ddc21-2a91-4165-b942-cf751f05852e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:24:12.407756 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.407682    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-maas-api-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:24:12.407756 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.407714    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b6g2\" (UniqueName: \"kubernetes.io/projected/5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7-kube-api-access-4b6g2\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:24:12.407756 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.407728    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1ddc21-2a91-4165-b942-cf751f05852e-metrics-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:24:12.407756 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.407740    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd986702-2d3c-4c48-b2f6-7b01417357d8-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:24:12.407756 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.407753    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bfdp\" (UniqueName: \"kubernetes.io/projected/fb1ddc21-2a91-4165-b942-cf751f05852e-kube-api-access-5bfdp\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:24:12.407997 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.407766    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:24:12.407997 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.407778    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4mzz\" (UniqueName: \"kubernetes.io/projected/cd986702-2d3c-4c48-b2f6-7b01417357d8-kube-api-access-k4mzz\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:24:12.538843 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.538813    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:12.639256 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:12.639208    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:13.050813 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.050777    2574 generic.go:356] "Generic (PLEG): container finished" podID="5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7" containerID="c3dc53d6174dad2c3751067e68d2f4fd2463a8f69d8590a2105e73247a802d03" exitCode=0
Sep 11 22:24:13.050813 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.050816    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"
Sep 11 22:24:13.051292 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.050859    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq" event={"ID":"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7","Type":"ContainerDied","Data":"c3dc53d6174dad2c3751067e68d2f4fd2463a8f69d8590a2105e73247a802d03"}
Sep 11 22:24:13.051292 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.050918    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq" event={"ID":"5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7","Type":"ContainerDied","Data":"425c9fa38e9d7e1080d597b716c16c58dd36d1220c04c81636e8f692251c477d"}
Sep 11 22:24:13.051292 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.050942    2574 scope.go:117] "RemoveContainer" containerID="c3dc53d6174dad2c3751067e68d2f4fd2463a8f69d8590a2105e73247a802d03"
Sep 11 22:24:13.052347 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.052305    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"
Sep 11 22:24:13.052347 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.052335    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47" event={"ID":"fb1ddc21-2a91-4165-b942-cf751f05852e","Type":"ContainerDied","Data":"091c1af288ab3c3753c15073767c66ba65f452b23e9312caf01c165b5a1258c1"}
Sep 11 22:24:13.053634 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.053620    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"
Sep 11 22:24:13.053758 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.053616    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl" event={"ID":"cd986702-2d3c-4c48-b2f6-7b01417357d8","Type":"ContainerDied","Data":"c361fc313f167a8ca64591a09f6c3a3fb9287a3f1c5f098897a60b7b49a26aab"}
Sep 11 22:24:13.060131 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.060109    2574 scope.go:117] "RemoveContainer" containerID="c3dc53d6174dad2c3751067e68d2f4fd2463a8f69d8590a2105e73247a802d03"
Sep 11 22:24:13.060477 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:24:13.060452    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3dc53d6174dad2c3751067e68d2f4fd2463a8f69d8590a2105e73247a802d03\": container with ID starting with c3dc53d6174dad2c3751067e68d2f4fd2463a8f69d8590a2105e73247a802d03 not found: ID does not exist" containerID="c3dc53d6174dad2c3751067e68d2f4fd2463a8f69d8590a2105e73247a802d03"
Sep 11 22:24:13.060562 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.060485    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dc53d6174dad2c3751067e68d2f4fd2463a8f69d8590a2105e73247a802d03"} err="failed to get container status \"c3dc53d6174dad2c3751067e68d2f4fd2463a8f69d8590a2105e73247a802d03\": rpc error: code = NotFound desc = could not find container \"c3dc53d6174dad2c3751067e68d2f4fd2463a8f69d8590a2105e73247a802d03\": container with ID starting with c3dc53d6174dad2c3751067e68d2f4fd2463a8f69d8590a2105e73247a802d03 not found: ID does not exist"
Sep 11 22:24:13.060562 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.060500    2574 scope.go:117] "RemoveContainer" containerID="5408e8ffd7056c68379c37cc8ef481ce6f0e86a122cd13d734dd8f9c433c2836"
Sep 11 22:24:13.069968 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.069954    2574 scope.go:117] "RemoveContainer" containerID="4f85788e3724326e812e1f4abc5ef96a8d37b161e4d9b775af45c08af043d7e9"
Sep 11 22:24:13.079098 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.079078    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"]
Sep 11 22:24:13.081245 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.081226    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6fe4c2-687645997c-lztbl"]
Sep 11 22:24:13.094320 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.094289    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"]
Sep 11 22:24:13.096641 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.096621    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6fe4c2-78c48d7b4b-zzgmq"]
Sep 11 22:24:13.107614 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.107594    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"]
Sep 11 22:24:13.111253 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.111235    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6fe4c2-79867bc76-ntv47"]
Sep 11 22:24:13.320606 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.320510    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:13.720694 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.720666    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7" path="/var/lib/kubelet/pods/5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7/volumes"
Sep 11 22:24:13.721078 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.721064    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd986702-2d3c-4c48-b2f6-7b01417357d8" path="/var/lib/kubelet/pods/cd986702-2d3c-4c48-b2f6-7b01417357d8/volumes"
Sep 11 22:24:13.721462 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:13.721448    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb1ddc21-2a91-4165-b942-cf751f05852e" path="/var/lib/kubelet/pods/fb1ddc21-2a91-4165-b942-cf751f05852e/volumes"
Sep 11 22:24:14.107595 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:14.107562    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:14.763409 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:14.763373    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:17.558879 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:17.558844    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:53.668214 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:53.668177    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:24:54.388607 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:24:54.388570    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:00.218217 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:00.218180    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:00.318484 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:00.318451    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:00.525604 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:00.525512    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:01.127667 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:01.127625    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:01.930726 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:01.930688    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:02.593229 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:02.593199    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:03.313515 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:03.313473    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:03.990506 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:03.990468    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:11.322758 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.322674    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"]
Sep 11 22:25:11.323147 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.322941    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5" podUID="7969118d-e553-4314-aacc-99dc73f011ac" containerName="maas-api" containerID="cri-o://bb21acf189550947864276758902574283c6dee7799563766627c54089f7bf5c" gracePeriod=30
Sep 11 22:25:11.350816 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.350790    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:11.385036 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.385004    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"]
Sep 11 22:25:11.385357 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.385242    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr" podUID="17c845ce-c7e7-4d72-930c-d7200d14175b" containerName="payload-processing" containerID="cri-o://ec467a8cf547ee0f78aa1170c5e6e14657a81b07d233d83ed57c4c59ade4e5ad" gracePeriod=30
Sep 11 22:25:11.434912 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.434879    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:11.548174 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.548123    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:11.610281 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.610201    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:11.639520 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.639496    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:25:11.667541 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.667519    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"
Sep 11 22:25:11.729378 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.729347    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-metrics-certs\") pod \"7969118d-e553-4314-aacc-99dc73f011ac\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") "
Sep 11 22:25:11.729513 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.729392    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-maas-api-tls\") pod \"7969118d-e553-4314-aacc-99dc73f011ac\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") "
Sep 11 22:25:11.729513 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.729420    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17c845ce-c7e7-4d72-930c-d7200d14175b-plugins-config-volume\") pod \"17c845ce-c7e7-4d72-930c-d7200d14175b\" (UID: \"17c845ce-c7e7-4d72-930c-d7200d14175b\") "
Sep 11 22:25:11.729513 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.729453    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmnnz\" (UniqueName: \"kubernetes.io/projected/7969118d-e553-4314-aacc-99dc73f011ac-kube-api-access-bmnnz\") pod \"7969118d-e553-4314-aacc-99dc73f011ac\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") "
Sep 11 22:25:11.729686 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.729525    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7969118d-e553-4314-aacc-99dc73f011ac-openshift-service-ca\") pod \"7969118d-e553-4314-aacc-99dc73f011ac\" (UID: \"7969118d-e553-4314-aacc-99dc73f011ac\") "
Sep 11 22:25:11.729686 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.729572    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjwhx\" (UniqueName: \"kubernetes.io/projected/17c845ce-c7e7-4d72-930c-d7200d14175b-kube-api-access-tjwhx\") pod \"17c845ce-c7e7-4d72-930c-d7200d14175b\" (UID: \"17c845ce-c7e7-4d72-930c-d7200d14175b\") "
Sep 11 22:25:11.729896 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.729848    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c845ce-c7e7-4d72-930c-d7200d14175b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17c845ce-c7e7-4d72-930c-d7200d14175b" (UID: "17c845ce-c7e7-4d72-930c-d7200d14175b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:11.730033 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.729905    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7969118d-e553-4314-aacc-99dc73f011ac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7969118d-e553-4314-aacc-99dc73f011ac" (UID: "7969118d-e553-4314-aacc-99dc73f011ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:11.731486 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.731458    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7969118d-e553-4314-aacc-99dc73f011ac" (UID: "7969118d-e553-4314-aacc-99dc73f011ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:25:11.731572 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.731513    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7969118d-e553-4314-aacc-99dc73f011ac" (UID: "7969118d-e553-4314-aacc-99dc73f011ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:25:11.731572 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.731529    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7969118d-e553-4314-aacc-99dc73f011ac-kube-api-access-bmnnz" (OuterVolumeSpecName: "kube-api-access-bmnnz") pod "7969118d-e553-4314-aacc-99dc73f011ac" (UID: "7969118d-e553-4314-aacc-99dc73f011ac"). InnerVolumeSpecName "kube-api-access-bmnnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:11.731953 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.731934    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c845ce-c7e7-4d72-930c-d7200d14175b-kube-api-access-tjwhx" (OuterVolumeSpecName: "kube-api-access-tjwhx") pod "17c845ce-c7e7-4d72-930c-d7200d14175b" (UID: "17c845ce-c7e7-4d72-930c-d7200d14175b"). InnerVolumeSpecName "kube-api-access-tjwhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:11.830789 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.830760    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7969118d-e553-4314-aacc-99dc73f011ac-openshift-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:25:11.830789 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.830785    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjwhx\" (UniqueName: \"kubernetes.io/projected/17c845ce-c7e7-4d72-930c-d7200d14175b-kube-api-access-tjwhx\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:25:11.830938 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.830800    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-metrics-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:25:11.830938 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.830815    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7969118d-e553-4314-aacc-99dc73f011ac-maas-api-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:25:11.830938 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.830826    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17c845ce-c7e7-4d72-930c-d7200d14175b-plugins-config-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:25:11.830938 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.830838    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmnnz\" (UniqueName: \"kubernetes.io/projected/7969118d-e553-4314-aacc-99dc73f011ac-kube-api-access-bmnnz\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:25:11.977826 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:11.977761    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:12.083282 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.083253    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:12.347918 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.347881    2574 generic.go:356] "Generic (PLEG): container finished" podID="7969118d-e553-4314-aacc-99dc73f011ac" containerID="bb21acf189550947864276758902574283c6dee7799563766627c54089f7bf5c" exitCode=0
Sep 11 22:25:12.347918 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.347907    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5" event={"ID":"7969118d-e553-4314-aacc-99dc73f011ac","Type":"ContainerDied","Data":"bb21acf189550947864276758902574283c6dee7799563766627c54089f7bf5c"}
Sep 11 22:25:12.348432 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.347926    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"
Sep 11 22:25:12.348432 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.347960    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5" event={"ID":"7969118d-e553-4314-aacc-99dc73f011ac","Type":"ContainerDied","Data":"2022715372e3858ed7f68a34f39da0bdb16f97e507483d60b7cd79f1e75f541f"}
Sep 11 22:25:12.348432 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.347983    2574 scope.go:117] "RemoveContainer" containerID="bb21acf189550947864276758902574283c6dee7799563766627c54089f7bf5c"
Sep 11 22:25:12.349465 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.349443    2574 generic.go:356] "Generic (PLEG): container finished" podID="17c845ce-c7e7-4d72-930c-d7200d14175b" containerID="ec467a8cf547ee0f78aa1170c5e6e14657a81b07d233d83ed57c4c59ade4e5ad" exitCode=0
Sep 11 22:25:12.349544 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.349472    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"
Sep 11 22:25:12.349544 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.349496    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr" event={"ID":"17c845ce-c7e7-4d72-930c-d7200d14175b","Type":"ContainerDied","Data":"ec467a8cf547ee0f78aa1170c5e6e14657a81b07d233d83ed57c4c59ade4e5ad"}
Sep 11 22:25:12.349544 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.349535    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr" event={"ID":"17c845ce-c7e7-4d72-930c-d7200d14175b","Type":"ContainerDied","Data":"e407e988b8d263c9ce60c95a06371fe71a8a535f303ea8257f4f40feeda1326a"}
Sep 11 22:25:12.358045 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.358024    2574 scope.go:117] "RemoveContainer" containerID="bb21acf189550947864276758902574283c6dee7799563766627c54089f7bf5c"
Sep 11 22:25:12.358359 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:25:12.358337    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb21acf189550947864276758902574283c6dee7799563766627c54089f7bf5c\": container with ID starting with bb21acf189550947864276758902574283c6dee7799563766627c54089f7bf5c not found: ID does not exist" containerID="bb21acf189550947864276758902574283c6dee7799563766627c54089f7bf5c"
Sep 11 22:25:12.358431 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.358368    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb21acf189550947864276758902574283c6dee7799563766627c54089f7bf5c"} err="failed to get container status \"bb21acf189550947864276758902574283c6dee7799563766627c54089f7bf5c\": rpc error: code = NotFound desc = could not find container \"bb21acf189550947864276758902574283c6dee7799563766627c54089f7bf5c\": container with ID starting with bb21acf189550947864276758902574283c6dee7799563766627c54089f7bf5c not found: ID does not exist"
Sep 11 22:25:12.358431 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.358384    2574 scope.go:117] "RemoveContainer" containerID="ec467a8cf547ee0f78aa1170c5e6e14657a81b07d233d83ed57c4c59ade4e5ad"
Sep 11 22:25:12.366642 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.366625    2574 scope.go:117] "RemoveContainer" containerID="ec467a8cf547ee0f78aa1170c5e6e14657a81b07d233d83ed57c4c59ade4e5ad"
Sep 11 22:25:12.366888 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:25:12.366868    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec467a8cf547ee0f78aa1170c5e6e14657a81b07d233d83ed57c4c59ade4e5ad\": container with ID starting with ec467a8cf547ee0f78aa1170c5e6e14657a81b07d233d83ed57c4c59ade4e5ad not found: ID does not exist" containerID="ec467a8cf547ee0f78aa1170c5e6e14657a81b07d233d83ed57c4c59ade4e5ad"
Sep 11 22:25:12.366936 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.366893    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec467a8cf547ee0f78aa1170c5e6e14657a81b07d233d83ed57c4c59ade4e5ad"} err="failed to get container status \"ec467a8cf547ee0f78aa1170c5e6e14657a81b07d233d83ed57c4c59ade4e5ad\": rpc error: code = NotFound desc = could not find container \"ec467a8cf547ee0f78aa1170c5e6e14657a81b07d233d83ed57c4c59ade4e5ad\": container with ID starting with ec467a8cf547ee0f78aa1170c5e6e14657a81b07d233d83ed57c4c59ade4e5ad not found: ID does not exist"
Sep 11 22:25:12.386031 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.386010    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"]
Sep 11 22:25:12.395621 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.395601    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbaf52-6749f5c495-lkxfr"]
Sep 11 22:25:12.420064 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.420034    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"]
Sep 11 22:25:12.425560 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.425536    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbaf52-c779588d7-7wmj5"]
Sep 11 22:25:12.835149 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:12.835030    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:13.721154 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:13.721123    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17c845ce-c7e7-4d72-930c-d7200d14175b" path="/var/lib/kubelet/pods/17c845ce-c7e7-4d72-930c-d7200d14175b/volumes"
Sep 11 22:25:13.721684 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:13.721663    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7969118d-e553-4314-aacc-99dc73f011ac" path="/var/lib/kubelet/pods/7969118d-e553-4314-aacc-99dc73f011ac/volumes"
Sep 11 22:25:13.728704 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:13.728667    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:18.005791 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.005748    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:18.105724 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.105688    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:18.266604 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.266505    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:18.641793 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.641749    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs"]
Sep 11 22:25:18.642215 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642196    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1ddc21-2a91-4165-b942-cf751f05852e" containerName="maas-api"
Sep 11 22:25:18.642299 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642216    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1ddc21-2a91-4165-b942-cf751f05852e" containerName="maas-api"
Sep 11 22:25:18.642299 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642239    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7" containerName="payload-pre-processing"
Sep 11 22:25:18.642299 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642248    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7" containerName="payload-pre-processing"
Sep 11 22:25:18.642299 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642272    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c845ce-c7e7-4d72-930c-d7200d14175b" containerName="payload-processing"
Sep 11 22:25:18.642299 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642281    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c845ce-c7e7-4d72-930c-d7200d14175b" containerName="payload-processing"
Sep 11 22:25:18.642593 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642304    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd986702-2d3c-4c48-b2f6-7b01417357d8" containerName="payload-processing"
Sep 11 22:25:18.642593 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642328    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd986702-2d3c-4c48-b2f6-7b01417357d8" containerName="payload-processing"
Sep 11 22:25:18.642593 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642341    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7969118d-e553-4314-aacc-99dc73f011ac" containerName="maas-api"
Sep 11 22:25:18.642593 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642349    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7969118d-e553-4314-aacc-99dc73f011ac" containerName="maas-api"
Sep 11 22:25:18.642593 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642357    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56f3ecf9-88fb-48eb-b30d-fa9c62dd4141" containerName="maas-api"
Sep 11 22:25:18.642593 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642366    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="56f3ecf9-88fb-48eb-b30d-fa9c62dd4141" containerName="maas-api"
Sep 11 22:25:18.642593 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642467    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a5d04e5-f8b1-4d9a-b89f-15a58ead08e7" containerName="payload-pre-processing"
Sep 11 22:25:18.642593 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642484    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd986702-2d3c-4c48-b2f6-7b01417357d8" containerName="payload-processing"
Sep 11 22:25:18.642593 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642496    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1ddc21-2a91-4165-b942-cf751f05852e" containerName="maas-api"
Sep 11 22:25:18.642593 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642508    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="17c845ce-c7e7-4d72-930c-d7200d14175b" containerName="payload-processing"
Sep 11 22:25:18.642593 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642516    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="56f3ecf9-88fb-48eb-b30d-fa9c62dd4141" containerName="maas-api"
Sep 11 22:25:18.642593 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.642526    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7969118d-e553-4314-aacc-99dc73f011ac" containerName="maas-api"
Sep 11 22:25:18.645767 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.645744    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs"
Sep 11 22:25:18.648667 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.648612    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:25:18.648667 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.648614    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:25:18.652557 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.652531    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs"]
Sep 11 22:25:18.688221 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.688190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb42551-e47d-4aae-8e2d-f6d4e0439915-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs\" (UID: \"7bb42551-e47d-4aae-8e2d-f6d4e0439915\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs"
Sep 11 22:25:18.788900 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.788863    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb42551-e47d-4aae-8e2d-f6d4e0439915-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs\" (UID: \"7bb42551-e47d-4aae-8e2d-f6d4e0439915\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs"
Sep 11 22:25:18.789495 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.789478    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb42551-e47d-4aae-8e2d-f6d4e0439915-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs\" (UID: \"7bb42551-e47d-4aae-8e2d-f6d4e0439915\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs"
Sep 11 22:25:18.896615 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.896512    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:18.957433 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:18.957398    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs"
Sep 11 22:25:19.233260 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:19.233238    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs"]
Sep 11 22:25:19.235264 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:25:19.235232    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb42551_e47d_4aae_8e2d_f6d4e0439915.slice/crio-c1476c2af884decd40e49d467f641dc21095fb4fdf632bc85be367ce70f88332 WatchSource:0}: Error finding container c1476c2af884decd40e49d467f641dc21095fb4fdf632bc85be367ce70f88332: Status 404 returned error can't find the container with id c1476c2af884decd40e49d467f641dc21095fb4fdf632bc85be367ce70f88332
Sep 11 22:25:19.385945 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:19.385905    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs" event={"ID":"7bb42551-e47d-4aae-8e2d-f6d4e0439915","Type":"ContainerStarted","Data":"c1476c2af884decd40e49d467f641dc21095fb4fdf632bc85be367ce70f88332"}
Sep 11 22:25:19.821596 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:19.821563    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:20.392498 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:20.392465    2574 generic.go:356] "Generic (PLEG): container finished" podID="7bb42551-e47d-4aae-8e2d-f6d4e0439915" containerID="ec139a720bfd1af4474700ed6b6a10b51a1548f9937d647a881a94f0f48f01ed" exitCode=0
Sep 11 22:25:20.392863 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:20.392536    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs" event={"ID":"7bb42551-e47d-4aae-8e2d-f6d4e0439915","Type":"ContainerDied","Data":"ec139a720bfd1af4474700ed6b6a10b51a1548f9937d647a881a94f0f48f01ed"}
Sep 11 22:25:20.474701 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:20.474669    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:21.212053 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:21.212017    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:21.528748 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:21.528723    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs"
Sep 11 22:25:21.612529 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:21.612497    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb42551-e47d-4aae-8e2d-f6d4e0439915-maas-api-service-ca\") pod \"7bb42551-e47d-4aae-8e2d-f6d4e0439915\" (UID: \"7bb42551-e47d-4aae-8e2d-f6d4e0439915\") "
Sep 11 22:25:21.612869 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:21.612844    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb42551-e47d-4aae-8e2d-f6d4e0439915-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7bb42551-e47d-4aae-8e2d-f6d4e0439915" (UID: "7bb42551-e47d-4aae-8e2d-f6d4e0439915"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:21.713983 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:21.713939    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb42551-e47d-4aae-8e2d-f6d4e0439915-maas-api-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:25:22.403725 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:22.403691    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs"
Sep 11 22:25:22.403940 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:22.403697    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs" event={"ID":"7bb42551-e47d-4aae-8e2d-f6d4e0439915","Type":"ContainerDied","Data":"c1476c2af884decd40e49d467f641dc21095fb4fdf632bc85be367ce70f88332"}
Sep 11 22:25:22.403940 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:22.403823    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1476c2af884decd40e49d467f641dc21095fb4fdf632bc85be367ce70f88332"
Sep 11 22:25:29.153677 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:29.153638    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:29.219056 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:29.219003    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:29.283904 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:29.283868    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:29.341833 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:29.341803    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs"]
Sep 11 22:25:29.344251 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:29.344230    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d09c96-hntfs"]
Sep 11 22:25:29.346106 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:29.346081    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:29.408450 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:29.408377    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:29.722232 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:29.722154    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb42551-e47d-4aae-8e2d-f6d4e0439915" path="/var/lib/kubelet/pods/7bb42551-e47d-4aae-8e2d-f6d4e0439915/volumes"
Sep 11 22:25:29.786792 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:29.786754    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:29.875821 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:29.875781    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:30.500102 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:30.500063    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:36.365370 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:36.365330    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:36.467903 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:36.467870    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:36.644580 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:36.644498    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:36.975234 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:36.975141    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66"]
Sep 11 22:25:36.975652 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:36.975632    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb42551-e47d-4aae-8e2d-f6d4e0439915" containerName="revoke-keys"
Sep 11 22:25:36.975763 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:36.975654    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb42551-e47d-4aae-8e2d-f6d4e0439915" containerName="revoke-keys"
Sep 11 22:25:36.975763 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:36.975758    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb42551-e47d-4aae-8e2d-f6d4e0439915" containerName="revoke-keys"
Sep 11 22:25:36.979107 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:36.979086    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66"
Sep 11 22:25:36.981833 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:36.981814    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:25:36.981912 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:36.981895    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:25:36.986878 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:36.986852    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66"]
Sep 11 22:25:37.154143 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:37.154101    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00ae607-018d-47a4-9a66-1fcb7d9cc3ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66\" (UID: \"a00ae607-018d-47a4-9a66-1fcb7d9cc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66"
Sep 11 22:25:37.255698 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:37.255596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00ae607-018d-47a4-9a66-1fcb7d9cc3ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66\" (UID: \"a00ae607-018d-47a4-9a66-1fcb7d9cc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66"
Sep 11 22:25:37.256275 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:37.256249    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00ae607-018d-47a4-9a66-1fcb7d9cc3ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66\" (UID: \"a00ae607-018d-47a4-9a66-1fcb7d9cc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66"
Sep 11 22:25:37.290568 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:37.290538    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66"
Sep 11 22:25:37.302239 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:37.302211    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:37.567329 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:37.567287    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66"]
Sep 11 22:25:37.569612 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:25:37.569582    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00ae607_018d_47a4_9a66_1fcb7d9cc3ff.slice/crio-c025bab40f8d23aa3c1c9704c8736364a527f1c7418c47f210721f30b6be6b1d WatchSource:0}: Error finding container c025bab40f8d23aa3c1c9704c8736364a527f1c7418c47f210721f30b6be6b1d: Status 404 returned error can't find the container with id c025bab40f8d23aa3c1c9704c8736364a527f1c7418c47f210721f30b6be6b1d
Sep 11 22:25:38.020631 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:38.020588    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:38.481520 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:38.481426    2574 generic.go:356] "Generic (PLEG): container finished" podID="a00ae607-018d-47a4-9a66-1fcb7d9cc3ff" containerID="7a90f8091a0b171ea422645806854418dac1b00d62a813a4ccaca3058a6971b5" exitCode=0
Sep 11 22:25:38.481520 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:38.481493    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66" event={"ID":"a00ae607-018d-47a4-9a66-1fcb7d9cc3ff","Type":"ContainerDied","Data":"7a90f8091a0b171ea422645806854418dac1b00d62a813a4ccaca3058a6971b5"}
Sep 11 22:25:38.481721 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:38.481541    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66" event={"ID":"a00ae607-018d-47a4-9a66-1fcb7d9cc3ff","Type":"ContainerStarted","Data":"c025bab40f8d23aa3c1c9704c8736364a527f1c7418c47f210721f30b6be6b1d"}
Sep 11 22:25:38.803770 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:38.803679    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:39.624488 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:39.624459    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66"
Sep 11 22:25:39.779354 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:39.779258    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00ae607-018d-47a4-9a66-1fcb7d9cc3ff-maas-api-service-ca\") pod \"a00ae607-018d-47a4-9a66-1fcb7d9cc3ff\" (UID: \"a00ae607-018d-47a4-9a66-1fcb7d9cc3ff\") "
Sep 11 22:25:39.779680 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:39.779653    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00ae607-018d-47a4-9a66-1fcb7d9cc3ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a00ae607-018d-47a4-9a66-1fcb7d9cc3ff" (UID: "a00ae607-018d-47a4-9a66-1fcb7d9cc3ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:39.880615 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:39.880580    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00ae607-018d-47a4-9a66-1fcb7d9cc3ff-maas-api-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:25:40.493503 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:40.493471    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66"
Sep 11 22:25:40.493694 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:40.493478    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66" event={"ID":"a00ae607-018d-47a4-9a66-1fcb7d9cc3ff","Type":"ContainerDied","Data":"c025bab40f8d23aa3c1c9704c8736364a527f1c7418c47f210721f30b6be6b1d"}
Sep 11 22:25:40.493694 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:40.493612    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c025bab40f8d23aa3c1c9704c8736364a527f1c7418c47f210721f30b6be6b1d"
Sep 11 22:25:47.420347 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:47.420279    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:47.488939 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:47.488892    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:47.586601 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:47.586554    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:47.602187 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:47.602156    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66"]
Sep 11 22:25:47.604771 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:47.604740    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-966527-jnt66"]
Sep 11 22:25:47.642915 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:47.642880    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:47.720166 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:47.720077    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a00ae607-018d-47a4-9a66-1fcb7d9cc3ff" path="/var/lib/kubelet/pods/a00ae607-018d-47a4-9a66-1fcb7d9cc3ff/volumes"
Sep 11 22:25:48.031980 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:48.031891    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:48.132852 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:48.132813    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:54.090125 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:54.090096    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:54.152431 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:54.152385    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:25:54.359594 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:25:54.359518    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:26:21.172626 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:26:21.172594    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:26:47.749256 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:26:47.749220    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:27:02.871321 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:27:02.871270    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:27:04.692433 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:27:04.692399    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:27:04.744870 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:27:04.744833    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:27:08.268253 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:27:08.268195    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:27:13.844354 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:27:13.844328    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:27:13.848732 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:27:13.848702    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:27:13.850684 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:27:13.850662    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:27:13.861144 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:27:13.861122    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:27:20.043499 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:27:20.043465    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:27:21.552575 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:27:21.552537    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:28:14.449123 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:14.449038    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:28:16.158329 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:16.158275    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:28:23.094134 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:23.094096    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"]
Sep 11 22:28:23.094643 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:23.094329    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc" podUID="385caa9e-eaad-4bdb-8a5d-d5697d286cce" containerName="manager" containerID="cri-o://dcd890be7164cf551b4a1904171a56a09c55e156fee1217798c4301d998ececf" gracePeriod=10
Sep 11 22:28:24.149485 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.149458    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"
Sep 11 22:28:24.221416 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.221322    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/385caa9e-eaad-4bdb-8a5d-d5697d286cce-extensions-socket-volume\") pod \"385caa9e-eaad-4bdb-8a5d-d5697d286cce\" (UID: \"385caa9e-eaad-4bdb-8a5d-d5697d286cce\") "
Sep 11 22:28:24.221581 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.221444    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpkrf\" (UniqueName: \"kubernetes.io/projected/385caa9e-eaad-4bdb-8a5d-d5697d286cce-kube-api-access-hpkrf\") pod \"385caa9e-eaad-4bdb-8a5d-d5697d286cce\" (UID: \"385caa9e-eaad-4bdb-8a5d-d5697d286cce\") "
Sep 11 22:28:24.221797 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.221772    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385caa9e-eaad-4bdb-8a5d-d5697d286cce-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "385caa9e-eaad-4bdb-8a5d-d5697d286cce" (UID: "385caa9e-eaad-4bdb-8a5d-d5697d286cce"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:28:24.223544 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.223518    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385caa9e-eaad-4bdb-8a5d-d5697d286cce-kube-api-access-hpkrf" (OuterVolumeSpecName: "kube-api-access-hpkrf") pod "385caa9e-eaad-4bdb-8a5d-d5697d286cce" (UID: "385caa9e-eaad-4bdb-8a5d-d5697d286cce"). InnerVolumeSpecName "kube-api-access-hpkrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:28:24.317129 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.317094    2574 generic.go:356] "Generic (PLEG): container finished" podID="385caa9e-eaad-4bdb-8a5d-d5697d286cce" containerID="dcd890be7164cf551b4a1904171a56a09c55e156fee1217798c4301d998ececf" exitCode=0
Sep 11 22:28:24.317302 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.317139    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"
Sep 11 22:28:24.317302 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.317152    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc" event={"ID":"385caa9e-eaad-4bdb-8a5d-d5697d286cce","Type":"ContainerDied","Data":"dcd890be7164cf551b4a1904171a56a09c55e156fee1217798c4301d998ececf"}
Sep 11 22:28:24.317302 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.317195    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc" event={"ID":"385caa9e-eaad-4bdb-8a5d-d5697d286cce","Type":"ContainerDied","Data":"ff42f4ba2ab5e9654f55c14e22141e26c3d94143eb93ecbdda414348ad8eae8d"}
Sep 11 22:28:24.317302 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.317216    2574 scope.go:117] "RemoveContainer" containerID="dcd890be7164cf551b4a1904171a56a09c55e156fee1217798c4301d998ececf"
Sep 11 22:28:24.322640 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.322613    2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/385caa9e-eaad-4bdb-8a5d-d5697d286cce-extensions-socket-volume\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:28:24.322768 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.322649    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpkrf\" (UniqueName: \"kubernetes.io/projected/385caa9e-eaad-4bdb-8a5d-d5697d286cce-kube-api-access-hpkrf\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:28:24.327867 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.327847    2574 scope.go:117] "RemoveContainer" containerID="dcd890be7164cf551b4a1904171a56a09c55e156fee1217798c4301d998ececf"
Sep 11 22:28:24.328212 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:28:24.328184    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd890be7164cf551b4a1904171a56a09c55e156fee1217798c4301d998ececf\": container with ID starting with dcd890be7164cf551b4a1904171a56a09c55e156fee1217798c4301d998ececf not found: ID does not exist" containerID="dcd890be7164cf551b4a1904171a56a09c55e156fee1217798c4301d998ececf"
Sep 11 22:28:24.328287 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.328225    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd890be7164cf551b4a1904171a56a09c55e156fee1217798c4301d998ececf"} err="failed to get container status \"dcd890be7164cf551b4a1904171a56a09c55e156fee1217798c4301d998ececf\": rpc error: code = NotFound desc = could not find container \"dcd890be7164cf551b4a1904171a56a09c55e156fee1217798c4301d998ececf\": container with ID starting with dcd890be7164cf551b4a1904171a56a09c55e156fee1217798c4301d998ececf not found: ID does not exist"
Sep 11 22:28:24.341861 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.341836    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"]
Sep 11 22:28:24.346938 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:24.346920    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6lgc"]
Sep 11 22:28:25.719597 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:28:25.719560    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="385caa9e-eaad-4bdb-8a5d-d5697d286cce" path="/var/lib/kubelet/pods/385caa9e-eaad-4bdb-8a5d-d5697d286cce/volumes"
Sep 11 22:29:23.810944 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.810901    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn"]
Sep 11 22:29:23.811497 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.811475    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385caa9e-eaad-4bdb-8a5d-d5697d286cce" containerName="manager"
Sep 11 22:29:23.811550 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.811501    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="385caa9e-eaad-4bdb-8a5d-d5697d286cce" containerName="manager"
Sep 11 22:29:23.811550 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.811519    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a00ae607-018d-47a4-9a66-1fcb7d9cc3ff" containerName="revoke-keys"
Sep 11 22:29:23.811550 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.811527    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a00ae607-018d-47a4-9a66-1fcb7d9cc3ff" containerName="revoke-keys"
Sep 11 22:29:23.811681 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.811658    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="385caa9e-eaad-4bdb-8a5d-d5697d286cce" containerName="manager"
Sep 11 22:29:23.811722 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.811684    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a00ae607-018d-47a4-9a66-1fcb7d9cc3ff" containerName="revoke-keys"
Sep 11 22:29:23.814879 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.814864    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn"
Sep 11 22:29:23.817540 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.817518    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 22:29:23.817669 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.817559    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 22:29:23.817821 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.817805    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8bxkr\""
Sep 11 22:29:23.825725 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.825703    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn"]
Sep 11 22:29:23.931391 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.931361    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q94mr\" (UniqueName: \"kubernetes.io/projected/c4b027de-d0fe-49e0-89e3-1cbb60e1ab59-kube-api-access-q94mr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p92nn\" (UID: \"c4b027de-d0fe-49e0-89e3-1cbb60e1ab59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn"
Sep 11 22:29:23.931535 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:23.931470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4b027de-d0fe-49e0-89e3-1cbb60e1ab59-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p92nn\" (UID: \"c4b027de-d0fe-49e0-89e3-1cbb60e1ab59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn"
Sep 11 22:29:24.031862 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:24.031827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q94mr\" (UniqueName: \"kubernetes.io/projected/c4b027de-d0fe-49e0-89e3-1cbb60e1ab59-kube-api-access-q94mr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p92nn\" (UID: \"c4b027de-d0fe-49e0-89e3-1cbb60e1ab59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn"
Sep 11 22:29:24.032028 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:24.031920    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4b027de-d0fe-49e0-89e3-1cbb60e1ab59-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p92nn\" (UID: \"c4b027de-d0fe-49e0-89e3-1cbb60e1ab59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn"
Sep 11 22:29:24.032284 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:24.032263    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4b027de-d0fe-49e0-89e3-1cbb60e1ab59-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p92nn\" (UID: \"c4b027de-d0fe-49e0-89e3-1cbb60e1ab59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn"
Sep 11 22:29:24.043967 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:24.043948    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q94mr\" (UniqueName: \"kubernetes.io/projected/c4b027de-d0fe-49e0-89e3-1cbb60e1ab59-kube-api-access-q94mr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p92nn\" (UID: \"c4b027de-d0fe-49e0-89e3-1cbb60e1ab59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn"
Sep 11 22:29:24.125488 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:24.125462    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn"
Sep 11 22:29:24.408954 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:24.408923    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn"]
Sep 11 22:29:24.409946 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:29:24.409922    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b027de_d0fe_49e0_89e3_1cbb60e1ab59.slice/crio-06853ab21f642a5d3577adc8be4236e1df344d9269964176f490164268d78c4d WatchSource:0}: Error finding container 06853ab21f642a5d3577adc8be4236e1df344d9269964176f490164268d78c4d: Status 404 returned error can't find the container with id 06853ab21f642a5d3577adc8be4236e1df344d9269964176f490164268d78c4d
Sep 11 22:29:24.611929 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:24.611893    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn" event={"ID":"c4b027de-d0fe-49e0-89e3-1cbb60e1ab59","Type":"ContainerStarted","Data":"7c552fe7ad4a7894d5f5262a44b3bfd58e8dc73349870fdb757714ab8380ebf5"}
Sep 11 22:29:24.612122 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:24.611947    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn"
Sep 11 22:29:24.612122 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:24.611967    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn" event={"ID":"c4b027de-d0fe-49e0-89e3-1cbb60e1ab59","Type":"ContainerStarted","Data":"06853ab21f642a5d3577adc8be4236e1df344d9269964176f490164268d78c4d"}
Sep 11 22:29:24.640489 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:24.635507    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn" podStartSLOduration=1.635485453 podStartE2EDuration="1.635485453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:29:24.631476855 +0000 UTC m=+2231.366588204" watchObservedRunningTime="2026-09-11 22:29:24.635485453 +0000 UTC m=+2231.370596801"
Sep 11 22:29:35.619459 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:29:35.619429    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p92nn"
Sep 11 22:30:01.460037 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:30:01.460004    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:30:10.404938 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:30:10.404904    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:30:29.378569 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:30:29.378538    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:30:48.916705 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:30:48.916667    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:30:49.712436 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:30:49.712398    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:31:07.013845 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.013758    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-758d76856b-k82xr"]
Sep 11 22:31:07.017512 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.017489    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.020327 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.020282    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:31:07.021745 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.021722    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 22:31:07.021884 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.021792    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 22:31:07.021884 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.021809    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mc9f9\""
Sep 11 22:31:07.021884 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.021844    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 22:31:07.027510 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.027486    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-758d76856b-k82xr"]
Sep 11 22:31:07.135944 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.135914    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npc7q\" (UniqueName: \"kubernetes.io/projected/d69e0039-1dc3-4a60-a904-b9d1d2379572-kube-api-access-npc7q\") pod \"maas-api-758d76856b-k82xr\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.136115 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.135968    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69e0039-1dc3-4a60-a904-b9d1d2379572-openshift-service-ca\") pod \"maas-api-758d76856b-k82xr\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.136115 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.136047    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d69e0039-1dc3-4a60-a904-b9d1d2379572-maas-api-tls\") pod \"maas-api-758d76856b-k82xr\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.136190 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.136118    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d69e0039-1dc3-4a60-a904-b9d1d2379572-metrics-certs\") pod \"maas-api-758d76856b-k82xr\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.237373 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.237338    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d69e0039-1dc3-4a60-a904-b9d1d2379572-metrics-certs\") pod \"maas-api-758d76856b-k82xr\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.237528 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.237389    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npc7q\" (UniqueName: \"kubernetes.io/projected/d69e0039-1dc3-4a60-a904-b9d1d2379572-kube-api-access-npc7q\") pod \"maas-api-758d76856b-k82xr\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.237528 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.237441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69e0039-1dc3-4a60-a904-b9d1d2379572-openshift-service-ca\") pod \"maas-api-758d76856b-k82xr\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.237528 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.237498    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d69e0039-1dc3-4a60-a904-b9d1d2379572-maas-api-tls\") pod \"maas-api-758d76856b-k82xr\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.238161 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.238140    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69e0039-1dc3-4a60-a904-b9d1d2379572-openshift-service-ca\") pod \"maas-api-758d76856b-k82xr\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.239890 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.239871    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d69e0039-1dc3-4a60-a904-b9d1d2379572-metrics-certs\") pod \"maas-api-758d76856b-k82xr\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.239971 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.239934    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d69e0039-1dc3-4a60-a904-b9d1d2379572-maas-api-tls\") pod \"maas-api-758d76856b-k82xr\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.245357 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.245338    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npc7q\" (UniqueName: \"kubernetes.io/projected/d69e0039-1dc3-4a60-a904-b9d1d2379572-kube-api-access-npc7q\") pod \"maas-api-758d76856b-k82xr\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.329563 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.329472    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:07.601181 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.601154    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-758d76856b-k82xr"]
Sep 11 22:31:07.603767 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:31:07.603740    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69e0039_1dc3_4a60_a904_b9d1d2379572.slice/crio-bd24671d36d456e84e0aab5cc924b67c7223a2cde5ae5366ecd4819a5af2a3cb WatchSource:0}: Error finding container bd24671d36d456e84e0aab5cc924b67c7223a2cde5ae5366ecd4819a5af2a3cb: Status 404 returned error can't find the container with id bd24671d36d456e84e0aab5cc924b67c7223a2cde5ae5366ecd4819a5af2a3cb
Sep 11 22:31:07.605485 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:07.605459    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:31:08.110777 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:08.110738    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr" event={"ID":"d69e0039-1dc3-4a60-a904-b9d1d2379572","Type":"ContainerStarted","Data":"1f4bbf56d462f8372308a246db4c30f93c8905ffd67640d256646ab82113e725"}
Sep 11 22:31:08.111235 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:08.110870    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:08.111235 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:08.110910    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr" event={"ID":"d69e0039-1dc3-4a60-a904-b9d1d2379572","Type":"ContainerStarted","Data":"bd24671d36d456e84e0aab5cc924b67c7223a2cde5ae5366ecd4819a5af2a3cb"}
Sep 11 22:31:08.128791 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:08.128734    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr" podStartSLOduration=1.705357991 podStartE2EDuration="2.128717671s" totalImagesPullingTime="423.35968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:31:08.127340231 +0000 UTC m=+2334.862451578" watchObservedRunningTime="2026-09-11 22:31:08.128717671 +0000 UTC m=+2334.863829016"
Sep 11 22:31:14.121820 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:14.121790    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:15.080641 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.080610    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"]
Sep 11 22:31:15.084022 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.084001    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.095165 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.095130    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"]
Sep 11 22:31:15.214484 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.214443    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/995d9916-2205-4084-91c3-88363b85bc26-maas-api-tls\") pod \"maas-api-6945bc8b6b-bc9gr\" (UID: \"995d9916-2205-4084-91c3-88363b85bc26\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.214891 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.214494    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc2cx\" (UniqueName: \"kubernetes.io/projected/995d9916-2205-4084-91c3-88363b85bc26-kube-api-access-kc2cx\") pod \"maas-api-6945bc8b6b-bc9gr\" (UID: \"995d9916-2205-4084-91c3-88363b85bc26\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.214891 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.214513    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/995d9916-2205-4084-91c3-88363b85bc26-openshift-service-ca\") pod \"maas-api-6945bc8b6b-bc9gr\" (UID: \"995d9916-2205-4084-91c3-88363b85bc26\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.214891 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.214648    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/995d9916-2205-4084-91c3-88363b85bc26-metrics-certs\") pod \"maas-api-6945bc8b6b-bc9gr\" (UID: \"995d9916-2205-4084-91c3-88363b85bc26\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.316211 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.316172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/995d9916-2205-4084-91c3-88363b85bc26-maas-api-tls\") pod \"maas-api-6945bc8b6b-bc9gr\" (UID: \"995d9916-2205-4084-91c3-88363b85bc26\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.316419 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.316222    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc2cx\" (UniqueName: \"kubernetes.io/projected/995d9916-2205-4084-91c3-88363b85bc26-kube-api-access-kc2cx\") pod \"maas-api-6945bc8b6b-bc9gr\" (UID: \"995d9916-2205-4084-91c3-88363b85bc26\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.316419 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.316242    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/995d9916-2205-4084-91c3-88363b85bc26-openshift-service-ca\") pod \"maas-api-6945bc8b6b-bc9gr\" (UID: \"995d9916-2205-4084-91c3-88363b85bc26\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.316419 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.316301    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/995d9916-2205-4084-91c3-88363b85bc26-metrics-certs\") pod \"maas-api-6945bc8b6b-bc9gr\" (UID: \"995d9916-2205-4084-91c3-88363b85bc26\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.316986 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.316960    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/995d9916-2205-4084-91c3-88363b85bc26-openshift-service-ca\") pod \"maas-api-6945bc8b6b-bc9gr\" (UID: \"995d9916-2205-4084-91c3-88363b85bc26\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.318734 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.318708    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/995d9916-2205-4084-91c3-88363b85bc26-metrics-certs\") pod \"maas-api-6945bc8b6b-bc9gr\" (UID: \"995d9916-2205-4084-91c3-88363b85bc26\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.318829 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.318738    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/995d9916-2205-4084-91c3-88363b85bc26-maas-api-tls\") pod \"maas-api-6945bc8b6b-bc9gr\" (UID: \"995d9916-2205-4084-91c3-88363b85bc26\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.323947 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.323928    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc2cx\" (UniqueName: \"kubernetes.io/projected/995d9916-2205-4084-91c3-88363b85bc26-kube-api-access-kc2cx\") pod \"maas-api-6945bc8b6b-bc9gr\" (UID: \"995d9916-2205-4084-91c3-88363b85bc26\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.396027 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.395991    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:15.664979 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:15.664953    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"]
Sep 11 22:31:15.666760 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:31:15.666736    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995d9916_2205_4084_91c3_88363b85bc26.slice/crio-07f0358c353c60b2d393cf94daa079608e236209763ee510ef8badba30c24053 WatchSource:0}: Error finding container 07f0358c353c60b2d393cf94daa079608e236209763ee510ef8badba30c24053: Status 404 returned error can't find the container with id 07f0358c353c60b2d393cf94daa079608e236209763ee510ef8badba30c24053
Sep 11 22:31:16.154708 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:16.154669    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr" event={"ID":"995d9916-2205-4084-91c3-88363b85bc26","Type":"ContainerStarted","Data":"caba8086f0d2ba718271ae9126ba4b1614dd80867de0b7ed63a4b9450fdc1f82"}
Sep 11 22:31:16.154903 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:16.154731    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr" event={"ID":"995d9916-2205-4084-91c3-88363b85bc26","Type":"ContainerStarted","Data":"07f0358c353c60b2d393cf94daa079608e236209763ee510ef8badba30c24053"}
Sep 11 22:31:16.154903 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:16.154762    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:16.171082 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:16.171031    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr" podStartSLOduration=0.857693541 podStartE2EDuration="1.17101257s" totalImagesPullingTime="313.319029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:31:16.170445742 +0000 UTC m=+2342.905557089" watchObservedRunningTime="2026-09-11 22:31:16.17101257 +0000 UTC m=+2342.906123917"
Sep 11 22:31:22.166839 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:22.166805    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-bc9gr"
Sep 11 22:31:22.218720 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:22.218683    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-758d76856b-k82xr"]
Sep 11 22:31:22.218986 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:22.218931    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr" podUID="d69e0039-1dc3-4a60-a904-b9d1d2379572" containerName="maas-api" containerID="cri-o://1f4bbf56d462f8372308a246db4c30f93c8905ffd67640d256646ab82113e725" gracePeriod=30
Sep 11 22:31:23.464439 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:23.464415    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:23.594160 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:23.594127    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69e0039-1dc3-4a60-a904-b9d1d2379572-openshift-service-ca\") pod \"d69e0039-1dc3-4a60-a904-b9d1d2379572\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") "
Sep 11 22:31:23.594329 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:23.594231    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d69e0039-1dc3-4a60-a904-b9d1d2379572-metrics-certs\") pod \"d69e0039-1dc3-4a60-a904-b9d1d2379572\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") "
Sep 11 22:31:23.594329 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:23.594256    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npc7q\" (UniqueName: \"kubernetes.io/projected/d69e0039-1dc3-4a60-a904-b9d1d2379572-kube-api-access-npc7q\") pod \"d69e0039-1dc3-4a60-a904-b9d1d2379572\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") "
Sep 11 22:31:23.594329 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:23.594276    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d69e0039-1dc3-4a60-a904-b9d1d2379572-maas-api-tls\") pod \"d69e0039-1dc3-4a60-a904-b9d1d2379572\" (UID: \"d69e0039-1dc3-4a60-a904-b9d1d2379572\") "
Sep 11 22:31:23.594586 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:23.594558    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69e0039-1dc3-4a60-a904-b9d1d2379572-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d69e0039-1dc3-4a60-a904-b9d1d2379572" (UID: "d69e0039-1dc3-4a60-a904-b9d1d2379572"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:31:23.596353 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:23.596322    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d69e0039-1dc3-4a60-a904-b9d1d2379572-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d69e0039-1dc3-4a60-a904-b9d1d2379572" (UID: "d69e0039-1dc3-4a60-a904-b9d1d2379572"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:31:23.596445 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:23.596373    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69e0039-1dc3-4a60-a904-b9d1d2379572-kube-api-access-npc7q" (OuterVolumeSpecName: "kube-api-access-npc7q") pod "d69e0039-1dc3-4a60-a904-b9d1d2379572" (UID: "d69e0039-1dc3-4a60-a904-b9d1d2379572"). InnerVolumeSpecName "kube-api-access-npc7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:31:23.596490 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:23.596462    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d69e0039-1dc3-4a60-a904-b9d1d2379572-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d69e0039-1dc3-4a60-a904-b9d1d2379572" (UID: "d69e0039-1dc3-4a60-a904-b9d1d2379572"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:31:23.695820 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:23.695793    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d69e0039-1dc3-4a60-a904-b9d1d2379572-metrics-certs\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:31:23.695820 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:23.695817    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npc7q\" (UniqueName: \"kubernetes.io/projected/d69e0039-1dc3-4a60-a904-b9d1d2379572-kube-api-access-npc7q\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:31:23.695954 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:23.695831    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d69e0039-1dc3-4a60-a904-b9d1d2379572-maas-api-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:31:23.695954 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:23.695843    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69e0039-1dc3-4a60-a904-b9d1d2379572-openshift-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\""
Sep 11 22:31:24.198383 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:24.198347    2574 generic.go:356] "Generic (PLEG): container finished" podID="d69e0039-1dc3-4a60-a904-b9d1d2379572" containerID="1f4bbf56d462f8372308a246db4c30f93c8905ffd67640d256646ab82113e725" exitCode=0
Sep 11 22:31:24.198545 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:24.198386    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr"
Sep 11 22:31:24.198545 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:24.198421    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr" event={"ID":"d69e0039-1dc3-4a60-a904-b9d1d2379572","Type":"ContainerDied","Data":"1f4bbf56d462f8372308a246db4c30f93c8905ffd67640d256646ab82113e725"}
Sep 11 22:31:24.198545 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:24.198467    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-758d76856b-k82xr" event={"ID":"d69e0039-1dc3-4a60-a904-b9d1d2379572","Type":"ContainerDied","Data":"bd24671d36d456e84e0aab5cc924b67c7223a2cde5ae5366ecd4819a5af2a3cb"}
Sep 11 22:31:24.198545 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:24.198489    2574 scope.go:117] "RemoveContainer" containerID="1f4bbf56d462f8372308a246db4c30f93c8905ffd67640d256646ab82113e725"
Sep 11 22:31:24.206859 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:24.206837    2574 scope.go:117] "RemoveContainer" containerID="1f4bbf56d462f8372308a246db4c30f93c8905ffd67640d256646ab82113e725"
Sep 11 22:31:24.207142 ip-10-0-128-201 kubenswrapper[2574]: E0911 22:31:24.207120    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4bbf56d462f8372308a246db4c30f93c8905ffd67640d256646ab82113e725\": container with ID starting with 1f4bbf56d462f8372308a246db4c30f93c8905ffd67640d256646ab82113e725 not found: ID does not exist" containerID="1f4bbf56d462f8372308a246db4c30f93c8905ffd67640d256646ab82113e725"
Sep 11 22:31:24.207217 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:24.207150    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4bbf56d462f8372308a246db4c30f93c8905ffd67640d256646ab82113e725"} err="failed to get container status \"1f4bbf56d462f8372308a246db4c30f93c8905ffd67640d256646ab82113e725\": rpc error: code = NotFound desc = could not find container \"1f4bbf56d462f8372308a246db4c30f93c8905ffd67640d256646ab82113e725\": container with ID starting with 1f4bbf56d462f8372308a246db4c30f93c8905ffd67640d256646ab82113e725 not found: ID does not exist"
Sep 11 22:31:24.215836 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:24.215811    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-758d76856b-k82xr"]
Sep 11 22:31:24.217687 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:24.217667    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-758d76856b-k82xr"]
Sep 11 22:31:25.720858 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:25.720822    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d69e0039-1dc3-4a60-a904-b9d1d2379572" path="/var/lib/kubelet/pods/d69e0039-1dc3-4a60-a904-b9d1d2379572/volumes"
Sep 11 22:31:39.807596 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:39.807561    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:31:43.643052 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:43.643023    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-x99wl_bd4bad2e-70df-4859-acb3-ea92c3183b12/server/0.log"
Sep 11 22:31:43.764646 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:43.764621    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-89ngx_36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2/manager/0.log"
Sep 11 22:31:44.156406 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:44.156378    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bf8845876-nclmg_ecc7b580-81a5-404f-a8f9-3cd655b40a23/copy-manifests/0.log"
Sep 11 22:31:44.163223 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:44.163202    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bf8845876-nclmg_ecc7b580-81a5-404f-a8f9-3cd655b40a23/manager/0.log"
Sep 11 22:31:44.511004 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:44.510929    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6945bc8b6b-bc9gr_995d9916-2205-4084-91c3-88363b85bc26/maas-api/0.log"
Sep 11 22:31:44.732354 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:44.732299    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819415-wrdw9_0b28c946-2d0d-4188-8858-cdaf94fa2353/cleanup/0.log"
Sep 11 22:31:46.240588 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:46.240555    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-p92nn_c4b027de-d0fe-49e0-89e3-1cbb60e1ab59/manager/0.log"
Sep 11 22:31:46.971304 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:46.971268    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-96pjh_4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7/discovery/0.log"
Sep 11 22:31:47.450390 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:47.450357    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-k5dpp_a78a736e-b63b-48f0-a7ee-b378f2723cb4/payload-pre-processing/0.log"
Sep 11 22:31:47.692388 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:47.692343    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-889c88f7b-28lxw_c667eac2-77fb-4708-a43f-ae76bcac6bc1/router/0.log"
Sep 11 22:31:48.153460 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:48.153433    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jw44h_c5790506-7835-4c5e-af3a-f5a4425d5814/main/0.log"
Sep 11 22:31:48.269328 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:48.269285    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-jprhs_7ddc4933-1d12-46e3-a34a-bc6935e9d01d/main/0.log"
Sep 11 22:31:48.619477 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:48.619450    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-r6qwn_f771b808-1542-49f5-b934-6d0ec3142608/main/0.log"
Sep 11 22:31:54.811612 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:54.811580    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:31:54.874155 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:54.874120    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log" limit=1024
Sep 11 22:31:57.615985 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:57.615941    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-628px_b5dd2da0-d2fc-47b4-8e68-f7d8d2bc3faa/global-pull-secret-syncer/0.log"
Sep 11 22:31:57.756183 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:57.756154    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-52qz9_39564d9a-8271-4a01-bfe2-8e7c30c227b2/konnectivity-agent/0.log"
Sep 11 22:31:57.863997 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:31:57.863968    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-201.ec2.internal_91e28972004d554b545a91ff30beb546/haproxy/0.log"
Sep 11 22:32:02.705531 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:02.705492    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-p92nn_c4b027de-d0fe-49e0-89e3-1cbb60e1ab59/manager/0.log"
Sep 11 22:32:04.644759 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:04.644725    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-h65fx_2eecf508-6314-4920-a2c6-4d90d43db253/cluster-monitoring-operator/0.log"
Sep 11 22:32:04.840746 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:04.840718    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-m582g_2466e0c1-64f6-4591-b3d2-cf3c50ece04c/monitoring-plugin/0.log"
Sep 11 22:32:04.965206 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:04.965123    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-br9dj_7571fe76-5cf1-447d-a61d-9d49e14657ea/node-exporter/0.log"
Sep 11 22:32:04.987748 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:04.987725    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-br9dj_7571fe76-5cf1-447d-a61d-9d49e14657ea/kube-rbac-proxy/0.log"
Sep 11 22:32:05.017761 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.017735    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-br9dj_7571fe76-5cf1-447d-a61d-9d49e14657ea/init-textfile/0.log"
Sep 11 22:32:05.268083 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.268001    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18f95c61-356a-4dc0-be36-13c25a5c5a0c/prometheus/0.log"
Sep 11 22:32:05.292051 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.292026    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18f95c61-356a-4dc0-be36-13c25a5c5a0c/config-reloader/0.log"
Sep 11 22:32:05.319480 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.319452    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18f95c61-356a-4dc0-be36-13c25a5c5a0c/thanos-sidecar/0.log"
Sep 11 22:32:05.346156 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.346133    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18f95c61-356a-4dc0-be36-13c25a5c5a0c/kube-rbac-proxy-web/0.log"
Sep 11 22:32:05.371283 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.371256    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18f95c61-356a-4dc0-be36-13c25a5c5a0c/kube-rbac-proxy/0.log"
Sep 11 22:32:05.396324 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.396292    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18f95c61-356a-4dc0-be36-13c25a5c5a0c/kube-rbac-proxy-thanos/0.log"
Sep 11 22:32:05.418816 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.418793    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18f95c61-356a-4dc0-be36-13c25a5c5a0c/init-config-reloader/0.log"
Sep 11 22:32:05.449665 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.449643    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-qt8pf_ab511a00-2391-40c9-a790-663015e8d534/prometheus-operator/0.log"
Sep 11 22:32:05.473177 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.473152    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-qt8pf_ab511a00-2391-40c9-a790-663015e8d534/kube-rbac-proxy/0.log"
Sep 11 22:32:05.637242 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.637212    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb545d8cf-xz62b_d094fef9-0e11-400b-aca0-ff688ec8ab79/thanos-query/0.log"
Sep 11 22:32:05.661143 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.661116    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb545d8cf-xz62b_d094fef9-0e11-400b-aca0-ff688ec8ab79/kube-rbac-proxy-web/0.log"
Sep 11 22:32:05.683863 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.683840    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb545d8cf-xz62b_d094fef9-0e11-400b-aca0-ff688ec8ab79/kube-rbac-proxy/0.log"
Sep 11 22:32:05.705943 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.705921    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb545d8cf-xz62b_d094fef9-0e11-400b-aca0-ff688ec8ab79/prom-label-proxy/0.log"
Sep 11 22:32:05.733202 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.733182    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb545d8cf-xz62b_d094fef9-0e11-400b-aca0-ff688ec8ab79/kube-rbac-proxy-rules/0.log"
Sep 11 22:32:05.755279 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:05.755255    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb545d8cf-xz62b_d094fef9-0e11-400b-aca0-ff688ec8ab79/kube-rbac-proxy-metrics/0.log"
Sep 11 22:32:07.753691 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.753648    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"]
Sep 11 22:32:07.754099 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.754031    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d69e0039-1dc3-4a60-a904-b9d1d2379572" containerName="maas-api"
Sep 11 22:32:07.754099 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.754041    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d69e0039-1dc3-4a60-a904-b9d1d2379572" containerName="maas-api"
Sep 11 22:32:07.754190 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.754104    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d69e0039-1dc3-4a60-a904-b9d1d2379572" containerName="maas-api"
Sep 11 22:32:07.757446 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.757426    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.760038 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.760008    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vbnb9\"/\"default-dockercfg-pz8hq\""
Sep 11 22:32:07.760148 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.760064    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbnb9\"/\"kube-root-ca.crt\""
Sep 11 22:32:07.760148 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.760061    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbnb9\"/\"openshift-service-ca.crt\""
Sep 11 22:32:07.766711 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.766688    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"]
Sep 11 22:32:07.792326 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.792291    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:32:07.796140 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.796124    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/2.log"
Sep 11 22:32:07.896260 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.896231    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f93c473-f6fb-4539-a5e3-dde9bf36b469-podres\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.896260 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.896262    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f93c473-f6fb-4539-a5e3-dde9bf36b469-proc\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.896441 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.896328    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f93c473-f6fb-4539-a5e3-dde9bf36b469-lib-modules\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.896441 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.896363    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f93c473-f6fb-4539-a5e3-dde9bf36b469-sys\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.896441 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.896398    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqvf4\" (UniqueName: \"kubernetes.io/projected/4f93c473-f6fb-4539-a5e3-dde9bf36b469-kube-api-access-lqvf4\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.997627 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.997596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqvf4\" (UniqueName: \"kubernetes.io/projected/4f93c473-f6fb-4539-a5e3-dde9bf36b469-kube-api-access-lqvf4\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.997755 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.997666    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f93c473-f6fb-4539-a5e3-dde9bf36b469-podres\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.997755 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.997688    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f93c473-f6fb-4539-a5e3-dde9bf36b469-proc\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.997755 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.997728    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f93c473-f6fb-4539-a5e3-dde9bf36b469-lib-modules\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.997899 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.997769    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f93c473-f6fb-4539-a5e3-dde9bf36b469-sys\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.997899 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.997796    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f93c473-f6fb-4539-a5e3-dde9bf36b469-proc\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.997899 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.997845    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f93c473-f6fb-4539-a5e3-dde9bf36b469-podres\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.997899 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.997857    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f93c473-f6fb-4539-a5e3-dde9bf36b469-sys\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:07.998059 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:07.997937    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f93c473-f6fb-4539-a5e3-dde9bf36b469-lib-modules\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:08.005704 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:08.005650    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqvf4\" (UniqueName: \"kubernetes.io/projected/4f93c473-f6fb-4539-a5e3-dde9bf36b469-kube-api-access-lqvf4\") pod \"perf-node-gather-daemonset-hzcgn\" (UID: \"4f93c473-f6fb-4539-a5e3-dde9bf36b469\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:08.067737 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:08.067705    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:08.343050 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:08.343020    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"]
Sep 11 22:32:08.346285 ip-10-0-128-201 kubenswrapper[2574]: W0911 22:32:08.346252    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4f93c473_f6fb_4539_a5e3_dde9bf36b469.slice/crio-6cdff8c67e09adcbdf7ec9593a8b40be5aa14e81ed58ee2bba1799bb6b098267 WatchSource:0}: Error finding container 6cdff8c67e09adcbdf7ec9593a8b40be5aa14e81ed58ee2bba1799bb6b098267: Status 404 returned error can't find the container with id 6cdff8c67e09adcbdf7ec9593a8b40be5aa14e81ed58ee2bba1799bb6b098267
Sep 11 22:32:08.425517 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:08.425483    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn" event={"ID":"4f93c473-f6fb-4539-a5e3-dde9bf36b469","Type":"ContainerStarted","Data":"6cdff8c67e09adcbdf7ec9593a8b40be5aa14e81ed58ee2bba1799bb6b098267"}
Sep 11 22:32:09.111569 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:09.111543    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-lvts5_1d26ad8d-4464-4bfc-981c-7fde3e3cdf48/volume-data-source-validator/0.log"
Sep 11 22:32:09.431944 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:09.431866    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn" event={"ID":"4f93c473-f6fb-4539-a5e3-dde9bf36b469","Type":"ContainerStarted","Data":"1a10991448f59137e9ed77f12d2a96bb08e3abbc75817c59c709d735befc5b76"}
Sep 11 22:32:09.432093 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:09.431976    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:09.449695 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:09.449651    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn" podStartSLOduration=2.449636934 podStartE2EDuration="2.449636934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 22:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:32:09.447151031 +0000 UTC m=+2396.182262376" watchObservedRunningTime="2026-09-11 22:32:09.449636934 +0000 UTC m=+2396.184748279"
Sep 11 22:32:10.286644 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:10.286618    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qlxmr_51337e48-d86f-4001-a58a-c6a2297fe84a/dns/0.log"
Sep 11 22:32:10.308149 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:10.308124    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qlxmr_51337e48-d86f-4001-a58a-c6a2297fe84a/kube-rbac-proxy/0.log"
Sep 11 22:32:10.428116 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:10.428094    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-js4ll_f1c80315-39cc-4397-be35-3d7de7f7e9a1/dns-node-resolver/0.log"
Sep 11 22:32:11.078839 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:11.078812    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vhjjf_fbd851a8-185e-4b99-bcb6-76c0d1c63f1c/node-ca/0.log"
Sep 11 22:32:12.506858 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:12.506828    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-96pjh_4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7/discovery/0.log"
Sep 11 22:32:12.618197 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:12.618168    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-k5dpp_a78a736e-b63b-48f0-a7ee-b378f2723cb4/payload-pre-processing/0.log"
Sep 11 22:32:12.681853 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:12.681823    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-889c88f7b-28lxw_c667eac2-77fb-4708-a43f-ae76bcac6bc1/router/0.log"
Sep 11 22:32:13.657452 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:13.657428    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k89mx_7198fa9e-17b2-42b7-b52a-09098330d6a0/serve-healthcheck-canary/0.log"
Sep 11 22:32:13.876476 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:13.876448    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:32:13.881098 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:13.881072    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:32:13.889232 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:13.889211    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8g8rn_6d54fe9c-b07b-4b7f-96ae-b0af3e131114/console-operator/1.log"
Sep 11 22:32:13.893240 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:13.893220    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzl98_f0cf56f5-b9bb-4eba-8d83-eca6c7178e13/ovn-acl-logging/0.log"
Sep 11 22:32:14.125994 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:14.125957    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-ckcpn_64b6b508-ccca-4b2f-8e17-44de2a0b6e55/insights-operator/0.log"
Sep 11 22:32:14.126849 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:14.126819    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-ckcpn_64b6b508-ccca-4b2f-8e17-44de2a0b6e55/insights-operator/1.log"
Sep 11 22:32:14.201928 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:14.201895    2574 scope.go:117] "RemoveContainer" containerID="ec139a720bfd1af4474700ed6b6a10b51a1548f9937d647a881a94f0f48f01ed"
Sep 11 22:32:14.209736 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:14.209720    2574 scope.go:117] "RemoveContainer" containerID="7a90f8091a0b171ea422645806854418dac1b00d62a813a4ccaca3058a6971b5"
Sep 11 22:32:14.221665 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:14.221648    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8gxh6_d92733da-a0e5-4341-aed9-ad49e2a6b4f5/kube-rbac-proxy/0.log"
Sep 11 22:32:14.244180 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:14.244160    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8gxh6_d92733da-a0e5-4341-aed9-ad49e2a6b4f5/exporter/0.log"
Sep 11 22:32:14.266330 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:14.266303    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8gxh6_d92733da-a0e5-4341-aed9-ad49e2a6b4f5/extractor/0.log"
Sep 11 22:32:15.446727 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:15.446697    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-hzcgn"
Sep 11 22:32:17.127155 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:17.127129    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-x99wl_bd4bad2e-70df-4859-acb3-ea92c3183b12/server/0.log"
Sep 11 22:32:17.221070 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:17.221041    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-89ngx_36f4d8f5-8315-4cbe-9ec2-30a21b0da9c2/manager/0.log"
Sep 11 22:32:17.444257 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:17.444231    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bf8845876-nclmg_ecc7b580-81a5-404f-a8f9-3cd655b40a23/manager/0.log"
Sep 11 22:32:17.469904 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:17.469879    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bf8845876-nclmg_ecc7b580-81a5-404f-a8f9-3cd655b40a23/copy-manifests/0.log"
Sep 11 22:32:18.974987 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:18.974958    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57494c65d5-wfxq5_a490e2da-66cd-43b4-9c38-78f639b34dd1/manager/0.log"
Sep 11 22:32:19.075362 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:19.075275    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-96pjh_4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7/discovery/0.log"
Sep 11 22:32:19.166255 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:19.166223    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-k5dpp_a78a736e-b63b-48f0-a7ee-b378f2723cb4/payload-pre-processing/0.log"
Sep 11 22:32:19.215698 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:19.215661    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-889c88f7b-28lxw_c667eac2-77fb-4708-a43f-ae76bcac6bc1/router/0.log"
Sep 11 22:32:21.160301 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:21.160268    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-96pjh_4af5c03f-e7b8-45fc-9f9b-cbf90511d3c7/discovery/0.log"
Sep 11 22:32:21.259564 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:21.259540    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-k5dpp_a78a736e-b63b-48f0-a7ee-b378f2723cb4/payload-pre-processing/0.log"
Sep 11 22:32:21.316448 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:21.316387    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-889c88f7b-28lxw_c667eac2-77fb-4708-a43f-ae76bcac6bc1/router/0.log"
Sep 11 22:32:24.560793 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:24.560749    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-xbc2g_90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a/kube-storage-version-migrator-operator/1.log"
Sep 11 22:32:24.561537 ip-10-0-128-201 kubenswrapper[2574]: I0911 22:32:24.561516    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-xbc2g_90f10a46-d1c8-43b4-b91a-5d8e8e47ec1a/kube-storage-version-migrator-operator/0.log"