Sep 09 10:26:00.223039 ip-10-0-128-193 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 10:26:00.615694 ip-10-0-128-193 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 10:26:00.615694 ip-10-0-128-193 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 10:26:00.615694 ip-10-0-128-193 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 10:26:00.615694 ip-10-0-128-193 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 10:26:00.615694 ip-10-0-128-193 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 10:26:00.616361 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.615754    2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 10:26:00.618563 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618547    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 10:26:00.618563 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618563    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618567    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618571    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618574    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618578    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618581    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618584    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618587    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618590    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618593    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618596    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618598    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618601    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618611    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618613    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618616    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618619    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618621    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618624    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618627    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 10:26:00.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618629    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618633    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618635    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618638    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618641    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618644    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618647    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618649    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618654    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618658    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618661    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618663    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618666    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618669    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618672    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618675    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618677    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618680    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618683    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618685    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 10:26:00.619148 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618688    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618690    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618692    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618695    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618697    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618700    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618702    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618704    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618707    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618709    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618717    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618720    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618723    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618725    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618728    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618731    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618734    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618736    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618739    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618742    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 10:26:00.619626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618744    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618746    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618750    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618754    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618758    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618761    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618778    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618783    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618786    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618788    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618793    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618795    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618798    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618801    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618804    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618808    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618812    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618815    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618817    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 10:26:00.620138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618820    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618823    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618826    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618829    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618832    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618834    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618837    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618842    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618845    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618847    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618850    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618853    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618855    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618858    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618860    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618863    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618866    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618868    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 10:26:00.620623 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.618871    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 10:26:00.622938 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622926    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 10:26:00.622938 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622937    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622940    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622943    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622946    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622956    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622959    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622961    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622965    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622967    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622971    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622973    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622976    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622979    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622982    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622984    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622987    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622989    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622993    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622995    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.622998    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 10:26:00.623000 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623000    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623003    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623006    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623009    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623012    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623014    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623017    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623020    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623022    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623025    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623027    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623030    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623032    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623035    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623038    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623040    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623043    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623046    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623048    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 10:26:00.623492 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623051    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623053    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623056    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623058    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623061    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623063    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623066    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623068    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623071    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623073    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623076    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623078    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623080    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623083    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623086    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623092    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623095    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623098    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623101    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623103    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 10:26:00.623980 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623106    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623109    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623111    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623114    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623116    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623119    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623122    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623124    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623127    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623129    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623132    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623135    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623139    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623143    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623146    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623149    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623152    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623154    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623157    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 10:26:00.624496 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623159    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623162    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623164    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623167    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623169    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623172    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623174    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623177    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623181    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623184    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623187    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623191    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623195    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623198    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623200    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623204    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623208    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623211    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623214    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 10:26:00.625061 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623217    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623566    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623570    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623573    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623577    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623580    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623583    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623586    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623589    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623591    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623594    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623596    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623599    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623601    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623604    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623608    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623611    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623614    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623617    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623619    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 10:26:00.625526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623622    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623626    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623628    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623631    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623634    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623637    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623640    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623642    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623645    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623647    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623650    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623652    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623655    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623657    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623660    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623662    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623665    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623668    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623671    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 10:26:00.626071 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623673    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623676    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623678    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623680    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623683    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623685    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623688    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623690    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623693    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623696    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623698    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623701    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623703    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623706    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623708    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623711    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623715    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623718    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623721    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623723    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 10:26:00.626552 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623726    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623728    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623731    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623734    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623736    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623739    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623741    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623743    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623746    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623748    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623751    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623754    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623756    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623759    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623776    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623781    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623784    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623787    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623790    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623793    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 10:26:00.627043 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623796    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623798    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623800    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623803    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623805    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623808    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623811    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623814    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623816    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623819    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623821    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623824    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623826    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623830    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623834    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623837    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623839    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623842    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623844    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 10:26:00.627550 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623847    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.623849    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.623920    2570 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.623955    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.623963    2570 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.623967    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.623972    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.623975    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.623979    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.623984    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.623987    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.623991    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.623994    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.623998    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624001    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624004    2570 flags.go:64] FLAG: --cgroup-root=""
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624008    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624011    2570 flags.go:64] FLAG: --client-ca-file=""
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624013    2570 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624017    2570 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624021    2570 flags.go:64] FLAG: --cluster-domain=""
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624024    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624027    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624030    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 10:26:00.628028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624033    2570 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624037    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624040    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624043    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624046    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624049    2570 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624052    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624055    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624059    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624062    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624068    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624071    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624073    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624076    2570 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624079    2570 flags.go:64] FLAG: --enable-server="true"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624083    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624087    2570 flags.go:64] FLAG: --event-burst="100"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624090    2570 flags.go:64] FLAG: --event-qps="50"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624093    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624096    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624099    2570 flags.go:64] FLAG: --eviction-hard=""
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624103    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624106    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624108    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624112    2570 flags.go:64] FLAG: --eviction-soft=""
Sep 09 10:26:00.628599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624114    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624117    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624122    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624125    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624128    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624131    2570 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624134    2570 flags.go:64] FLAG: --feature-gates=""
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624138    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624141    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624144    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624147    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624150    2570 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624153    2570 flags.go:64] FLAG: --help="false"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624156    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-128-193.ec2.internal"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624159    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624162    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624170    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624175    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624178    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624181    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624184    2570 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624187    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624190    2570 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 10:26:00.629215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624194    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624197    2570 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624200    2570 flags.go:64] FLAG: --kube-reserved=""
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624203    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624206    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624209    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624212    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624215    2570 flags.go:64] FLAG: --lock-file=""
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624218    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624221    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624224    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624231    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624234    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624236    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624240    2570 flags.go:64] FLAG: --logging-format="text"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624243    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624246    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624249    2570 flags.go:64] FLAG: --manifest-url=""
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624252    2570 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624256    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624259    2570 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624263    2570 flags.go:64] FLAG: --max-pods="110"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624266    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624269    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624272    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 10:26:00.629810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624275    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624280    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624283    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624286    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624306    2570 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624310    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624314    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624317    2570 flags.go:64] FLAG: --pod-cidr=""
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624320    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624326    2570 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624329    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624332    2570 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624335    2570 flags.go:64] FLAG: --port="10250"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624338    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624341    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-017e191167a9d0e1c"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624344    2570 flags.go:64] FLAG: --qos-reserved=""
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624347    2570 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624350    2570 flags.go:64] FLAG: --register-node="true"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624353    2570 flags.go:64] FLAG: --register-with-taints=""
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624359    2570 flags.go:64] FLAG: --registry-burst="10"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624361    2570 flags.go:64] FLAG: --registry-qps="5"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624364    2570 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624368    2570 flags.go:64] FLAG: --reserved-memory=""
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624371    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624374    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 10:26:00.630423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624377    2570 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624380    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624383    2570 flags.go:64] FLAG: --runonce="false"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624386    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624389    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624392    2570 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624395    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624398    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624403    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624406    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624409    2570 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624412    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624415    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624418    2570 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624420    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624423    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624427    2570 flags.go:64] FLAG: --system-cgroups=""
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624430    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624436    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624439    2570 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624441    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624445    2570 flags.go:64] FLAG: --tls-min-version=""
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624448    2570 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624451    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624454    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 10:26:00.631545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624464    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624469    2570 flags.go:64] FLAG: --v="2"
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624473    2570 flags.go:64] FLAG: --version="false"
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624477    2570 flags.go:64] FLAG: --vmodule=""
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624481    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624484    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624599    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624603    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624606    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624609    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624612    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624615    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624617    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624620    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624623    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624627    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624630    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624632    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624635    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624638    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624641    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 10:26:00.632613 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624643    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624646    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624648    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624652    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624654    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624657    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624659    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624662    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624665    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624668    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624670    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624673    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624679    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624682    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624684    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624687    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624689    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624692    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624694    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624697    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 10:26:00.633220 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624700    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624703    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624705    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624708    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624710    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624713    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624716    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624719    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624721    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624724    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624727    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624729    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624731    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624734    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624736    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624739    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624742    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624745    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624748    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624751    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 10:26:00.633732 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624753    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624756    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624758    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624777    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624783    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624786    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624789    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624791    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624794    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624797    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624799    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624802    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624804    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624807    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624811    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624814    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624817    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624820    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624824    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624827    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 10:26:00.634246 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624830    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624834    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624838    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624843    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624845    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624848    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624851    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624854    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624858    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624865    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624868    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624871    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624874    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624877    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624880    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624882    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624886    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624889    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624892    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 10:26:00.634754 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624895    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624897    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624900    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624903    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.624906    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.624911    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.632664    2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.632686    2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.634725    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.634741    2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634841    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634851    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634857    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634862    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634867    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 10:26:00.635653 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634871    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634875    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634880    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634884    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634889    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634894    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634898    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634902    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634906    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634911    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634915    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634919    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634923    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634927    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634932    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634936    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634940    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634945    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634949    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 10:26:00.636278 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634954    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634958    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634962    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634966    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634970    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634977    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634981    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634986    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634991    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634995    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.634999    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635003    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635007    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635011    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635015    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635019    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635024    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635028    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635032    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635036    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 10:26:00.636946 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635043    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635050    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635055    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635060    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635065    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635069    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635075    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635082    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635087    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635093    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635097    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635101    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635106    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635111    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635116    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635120    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635124    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635129    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635134    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 10:26:00.637801 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635139    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635142    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635147    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635151    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635155    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635159    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635163    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635167    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635171    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635175    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635179    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635183    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635187    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635191    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635195    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635199    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635204    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635208    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635212    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635216    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 10:26:00.638429 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635220    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635224    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635228    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635232    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635236    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635240    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635244    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635249    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635253    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635257    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635262    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635267    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635272    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635276    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635280    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 10:26:00.639066 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635284    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.635292    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635468    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635476    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635483    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635489    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635495    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635499    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635504    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635508    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635512    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635516    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635521    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635525    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 10:26:00.639439 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635529    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635533    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635537    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635541    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635545    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635549    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635554    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635558    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635562    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635567    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635572    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635578    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635582    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635587    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635591    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635596    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635601    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635605    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635609    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635613    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 10:26:00.639846 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635617    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635622    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635626    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635630    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635634    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635638    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635643    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635647    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635650    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635654    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635658    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635662    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635666    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635670    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635674    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635678    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635682    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635686    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635690    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635694    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 10:26:00.640334 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635698    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635702    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635707    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635711    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635715    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635719    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635723    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635728    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635732    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635737    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635742    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635746    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635750    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635754    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635760    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635786    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635791    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635796    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635802    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 10:26:00.640998 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635806    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635810    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635815    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635820    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635825    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635830    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635834    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635839    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635844    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635848    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635853    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635857    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635861    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635865    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635869    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635873    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635877    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635881    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635888    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 10:26:00.641476 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635894    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635898    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635902    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635906    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635912    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635916    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635920    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635924    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:00.635929    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.635936    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.636090    2570 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.638693    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.639750    2570 server.go:1019] "Starting client certificate rotation"
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.639869    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 10:26:00.642185 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.639925    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 10:26:00.642736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.642318    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 10:26:00.646946 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.646921    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 10:26:00.657798 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.657779    2570 log.go:25] "Validated CRI v1 runtime API"
Sep 09 10:26:00.660529 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.660514    2570 log.go:25] "Validated CRI v1 image API"
Sep 09 10:26:00.662576 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.662564    2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 10:26:00.664882 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.664860    2570 fs.go:135] Filesystem UUIDs: map[6844b9f2-3fa3-4e38-b817-ffdccb3586c1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 edfd6baa-970d-4ae7-8fcf-a0e9d8278f29:/dev/nvme0n1p4]
Sep 09 10:26:00.664925 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.664883    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 10:26:00.670667 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.670558    2570 manager.go:216] Machine: {Timestamp:2026-09-09 10:26:00.669115004 +0000 UTC m=+0.332753970 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099767 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c5078decf6fc5fe6aab0ec273087d SystemUUID:ec2c5078-decf-6fc5-fe6a-ab0ec273087d BootID:c649fe2e-aed8-4df2-be1a-d66dd88cbd27 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9c:6b:c0:db:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9c:6b:c0:db:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:ae:0b:25:50:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 10:26:00.670667 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.670663    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 10:26:00.670792 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.670743    2570 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 10:26:00.673549 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.673527    2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 10:26:00.673679 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.673552    2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-193.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 10:26:00.674319 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.674310    2570 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 10:26:00.674365 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.674321    2570 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 10:26:00.674365 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.674334    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 10:26:00.674875 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.674865    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 10:26:00.676141 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.676132    2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 10:26:00.680447 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.680434    2570 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 10:26:00.680566 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.680556    2570 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 10:26:00.680599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.680585    2570 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 10:26:00.680599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.680595    2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 10:26:00.680678 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.680609    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 10:26:00.680678 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.680618    2570 kubelet.go:398] "Adding apiserver pod source"
Sep 09 10:26:00.680678 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.680626    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 10:26:00.681040 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.681014    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 10:26:00.681040 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.681035    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 10:26:00.682736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.682721    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 10:26:00.683342 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.683328    2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 10:26:00.685675 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.685660    2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 10:26:00.685760 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.685697    2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 10:26:00.685948 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.685936    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 10:26:00.686009 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.685976    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 10:26:00.686009 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.685988    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 10:26:00.686009 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.685996    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 10:26:00.686009 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.686008    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 10:26:00.686009 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.686017    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 10:26:00.686672 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.686661    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 10:26:00.686725 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.686674    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 10:26:00.686725 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.686684    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 10:26:00.686725 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.686693    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 10:26:00.686725 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.686711    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 10:26:00.686725 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.686723    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 10:26:00.687523 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.687512    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 10:26:00.687576 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.687525    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 10:26:00.691444 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.691429    2570 server.go:1268] "Started kubelet"
Sep 09 10:26:00.691547 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.691512    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 10:26:00.691632 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.691572    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 10:26:00.692073 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.692044    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 10:26:00.692446 ip-10-0-128-193 systemd[1]: Started Kubernetes Kubelet.
Sep 09 10:26:00.694917 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.694897    2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 10:26:00.695301 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.695281    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 10:26:00.695408 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.695349    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-193.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 10:26:00.695464 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.695396    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 10:26:00.698183 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.698168    2570 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 10:26:00.699391 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.699368    2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 10:26:00.699986 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.698639    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-193.ec2.internal.18d3a025ae1a287b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-193.ec2.internal,UID:ip-10-0-128-193.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-193.ec2.internal,},FirstTimestamp:2026-09-09 10:26:00.691402875 +0000 UTC m=+0.355041840,LastTimestamp:2026-09-09 10:26:00.691402875 +0000 UTC m=+0.355041840,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-193.ec2.internal,}"
Sep 09 10:26:00.703329 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.703313    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 10:26:00.703329 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.703320    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 10:26:00.703454 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.703373    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:00.703952 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.703936    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 10:26:00.704060 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.703938    2570 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 10:26:00.704060 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.704062    2570 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 10:26:00.704190 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.704173    2570 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 10:26:00.704190 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.704185    2570 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 10:26:00.704267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.704210    2570 factory.go:55] Registering systemd factory
Sep 09 10:26:00.704267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.704225    2570 factory.go:223] Registration of the systemd container factory successfully
Sep 09 10:26:00.704431 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.704417    2570 factory.go:153] Registering CRI-O factory
Sep 09 10:26:00.704431 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.704433    2570 factory.go:223] Registration of the crio container factory successfully
Sep 09 10:26:00.704543 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.704493    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 10:26:00.704543 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.704514    2570 factory.go:103] Registering Raw factory
Sep 09 10:26:00.704543 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.704528    2570 manager.go:1196] Started watching for new ooms in manager
Sep 09 10:26:00.705027 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.705008    2570 manager.go:349] Starting recovery of all containers
Sep 09 10:26:00.713326 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.713282    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 10:26:00.715199 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.715170    2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 10:26:00.715572 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.715544    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 10:26:00.717310 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.716327    2570 manager.go:354] Recovery completed
Sep 09 10:26:00.719238 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.719215    2570 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 09 10:26:00.722251 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.722237    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:26:00.724483 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.724446    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:26:00.724545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.724500    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:26:00.724545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.724515    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:26:00.725076 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.725063    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 10:26:00.725076 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.725076    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 10:26:00.725153 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.725091    2570 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 10:26:00.725153 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.725068    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:26:00.725934 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.725919    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:26:00.725979 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.725949    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:26:00.725979 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.725960    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:26:00.727245 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.727230    2570 policy_none.go:49] "None policy: Start"
Sep 09 10:26:00.727245 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.727252    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 10:26:00.727381 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.727266    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 10:26:00.727428 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.727326    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-193.ec2.internal.18d3a025b012dfbe  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-193.ec2.internal,UID:ip-10-0-128-193.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-193.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-193.ec2.internal,},FirstTimestamp:2026-09-09 10:26:00.724479934 +0000 UTC m=+0.388118899,LastTimestamp:2026-09-09 10:26:00.724479934 +0000 UTC m=+0.388118899,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-193.ec2.internal,}"
Sep 09 10:26:00.729275 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.729260    2570 policy_none.go:47] "Start"
Sep 09 10:26:00.737417 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.737299    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-193.ec2.internal.18d3a025b013547b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-193.ec2.internal,UID:ip-10-0-128-193.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-193.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-193.ec2.internal,},FirstTimestamp:2026-09-09 10:26:00.724509819 +0000 UTC m=+0.388148783,LastTimestamp:2026-09-09 10:26:00.724509819 +0000 UTC m=+0.388148783,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-193.ec2.internal,}"
Sep 09 10:26:00.740186 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.740116    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-193.ec2.internal.18d3a025b0137aeb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-193.ec2.internal,UID:ip-10-0-128-193.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-193.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-193.ec2.internal,},FirstTimestamp:2026-09-09 10:26:00.724519659 +0000 UTC m=+0.388158623,LastTimestamp:2026-09-09 10:26:00.724519659 +0000 UTC m=+0.388158623,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-193.ec2.internal,}"
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.754376    2570 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-193.ec2.internal.18d3a025b012dfbe\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-193.ec2.internal.18d3a025b012dfbe  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-193.ec2.internal,UID:ip-10-0-128-193.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-193.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-193.ec2.internal,},FirstTimestamp:2026-09-09 10:26:00.724479934 +0000 UTC m=+0.388118899,LastTimestamp:2026-09-09 10:26:00.725934628 +0000 UTC m=+0.389573595,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-193.ec2.internal,}"
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.756953    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ftqg9"
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.765308    2570 manager.go:337] "Starting Device Plugin manager"
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.765339    2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.765348    2570 server.go:87] "Starting device plugin registration server"
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.765544    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.765556    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.765731    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.765810    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.765819    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.766382    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.766433    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.770778    2570 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-193.ec2.internal.18d3a025b013547b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-193.ec2.internal.18d3a025b013547b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-193.ec2.internal,UID:ip-10-0-128-193.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-193.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-193.ec2.internal,},FirstTimestamp:2026-09-09 10:26:00.724509819 +0000 UTC m=+0.388148783,LastTimestamp:2026-09-09 10:26:00.725954854 +0000 UTC m=+0.389593819,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-193.ec2.internal,}"
Sep 09 10:26:00.780360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.772801    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ftqg9"
Sep 09 10:26:00.832800 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.832759    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 10:26:00.832916 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.832823    2570 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 10:26:00.832916 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.832845    2570 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 10:26:00.833005 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.832924    2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 10:26:00.836181 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.836150    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 10:26:00.866002 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.865952    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:26:00.866899 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.866883    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:26:00.866975 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.866911    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:26:00.866975 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.866921    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:26:00.866975 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.866942    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-193.ec2.internal"
Sep 09 10:26:00.874907 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.874891    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-193.ec2.internal"
Sep 09 10:26:00.874942 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.874914    2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-193.ec2.internal\": node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:00.889176 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.889154    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:00.933841 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.933814    2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-193.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal"]
Sep 09 10:26:00.933928 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.933876    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:26:00.935400 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.935385    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:26:00.935487 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.935417    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:26:00.935487 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.935431    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:26:00.937565 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.937550    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:26:00.937659 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.937643    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:00.937708 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.937669    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:26:00.938291 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.938276    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:26:00.938291 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.938283    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:26:00.938399 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.938306    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:26:00.938399 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.938321    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:26:00.938399 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.938308    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:26:00.938399 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.938394    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:26:00.940422 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.940406    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:00.940524 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.940429    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:26:00.941095 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.941078    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:26:00.941187 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.941105    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:26:00.941187 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:00.941114    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:26:00.968341 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.968322    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-193.ec2.internal\" not found" node="ip-10-0-128-193.ec2.internal"
Sep 09 10:26:00.972700 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.972686    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-193.ec2.internal\" not found" node="ip-10-0-128-193.ec2.internal"
Sep 09 10:26:00.989807 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:00.989789    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:01.004750 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.004722    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a4115943f85345d07176841ce75f166-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal\" (UID: \"9a4115943f85345d07176841ce75f166\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:01.004840 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.004758    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/046071a0f74eb379e298ca5ffe8a80f1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-193.ec2.internal\" (UID: \"046071a0f74eb379e298ca5ffe8a80f1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:01.004840 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.004807    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a4115943f85345d07176841ce75f166-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal\" (UID: \"9a4115943f85345d07176841ce75f166\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:01.090502 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:01.090476    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:01.105067 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.105047    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a4115943f85345d07176841ce75f166-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal\" (UID: \"9a4115943f85345d07176841ce75f166\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:01.105142 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.105079    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a4115943f85345d07176841ce75f166-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal\" (UID: \"9a4115943f85345d07176841ce75f166\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:01.105142 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.105100    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/046071a0f74eb379e298ca5ffe8a80f1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-193.ec2.internal\" (UID: \"046071a0f74eb379e298ca5ffe8a80f1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:01.105216 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.105150    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/046071a0f74eb379e298ca5ffe8a80f1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-193.ec2.internal\" (UID: \"046071a0f74eb379e298ca5ffe8a80f1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:01.105216 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.105199    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a4115943f85345d07176841ce75f166-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal\" (UID: \"9a4115943f85345d07176841ce75f166\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:01.105287 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.105223    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a4115943f85345d07176841ce75f166-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal\" (UID: \"9a4115943f85345d07176841ce75f166\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:01.191058 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:01.190981    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:01.271443 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.271407    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:01.274970 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.274941    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:01.291399 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:01.291261    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:01.391744 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:01.391705    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:01.494623 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:01.494536    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:01.601926 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:01.601897    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:01.639373 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.639333    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 10:26:01.640059 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.639539    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 10:26:01.682018 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.681986    2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 10:26:01.702436 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:01.702415    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:01.775957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.775872    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 10:21:01 +0000 UTC" deadline="2028-06-20 01:11:54.209146028 +0000 UTC"
Sep 09 10:26:01.775957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.775922    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15590h45m52.433227967s"
Sep 09 10:26:01.804157 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:01.804132    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:01.864922 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.864878    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:26:01.866046 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.866024    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:26:01.866160 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.866060    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:26:01.866160 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.866071    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:26:01.904690 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:01.904667    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-193.ec2.internal\" not found"
Sep 09 10:26:01.970294 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:01.970268    2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 10:26:02.003812 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.003759    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:02.025035 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.025011    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 10:26:02.026749 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.026683    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal"
Sep 09 10:26:02.054030 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.054003    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 10:26:02.109717 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:02.109665    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046071a0f74eb379e298ca5ffe8a80f1.slice/crio-a6a8f8e9cb2ba9f9ed6633adad8d8f328c8c237c27590fb63254f0250f3d1402 WatchSource:0}: Error finding container a6a8f8e9cb2ba9f9ed6633adad8d8f328c8c237c27590fb63254f0250f3d1402: Status 404 returned error can't find the container with id a6a8f8e9cb2ba9f9ed6633adad8d8f328c8c237c27590fb63254f0250f3d1402
Sep 09 10:26:02.114844 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.114829    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 10:26:02.241138 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.241112    2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 10:26:02.265268 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:02.265239    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4115943f85345d07176841ce75f166.slice/crio-f8a1af195bd33f42c294742bc9f30858cd1f0d08ea350eaf84e660f537a993c5 WatchSource:0}: Error finding container f8a1af195bd33f42c294742bc9f30858cd1f0d08ea350eaf84e660f537a993c5: Status 404 returned error can't find the container with id f8a1af195bd33f42c294742bc9f30858cd1f0d08ea350eaf84e660f537a993c5
Sep 09 10:26:02.682024 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.681979    2570 apiserver.go:52] "Watching apiserver"
Sep 09 10:26:02.692781 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.692747    2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 10:26:02.695695 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.695665    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-dh892","openshift-multus/multus-additional-cni-plugins-vlj8h","openshift-multus/network-metrics-daemon-llgq5","openshift-network-operator/iptables-alerter-qpgmv","kube-system/global-pull-secret-syncer-crdlr","kube-system/konnectivity-agent-4t6qk","openshift-cluster-node-tuning-operator/tuned-sh98r","openshift-image-registry/node-ca-5j47k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal","openshift-multus/multus-4qlxw","openshift-network-diagnostics/network-check-target-rsgg6","openshift-ovn-kubernetes/ovnkube-node-qs8qn","kube-system/kube-apiserver-proxy-ip-10-0-128-193.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"]
Sep 09 10:26:02.698675 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.698642    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.700824 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.700799    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.703333 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.703308    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.703631 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.703610    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 10:26:02.704565 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.704545    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 10:26:02.704724 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.704707    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 10:26:02.704993 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.704975    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-txrf8\""
Sep 09 10:26:02.705753 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.705731    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qpgmv"
Sep 09 10:26:02.708257 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.708238    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5j47k"
Sep 09 10:26:02.710743 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.710724    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4t6qk"
Sep 09 10:26:02.711589 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.711565    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 10:26:02.711698 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.711597    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 10:26:02.711824 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.711806    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 10:26:02.711907 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.711824    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 10:26:02.711907 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.711851    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 10:26:02.712282 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712264    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rplc9\""
Sep 09 10:26:02.712392 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712290    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8g9hk\""
Sep 09 10:26:02.712392 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712302    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 10:26:02.712392 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712295    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 10:26:02.712392 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712299    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-modprobe-d\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.712588 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712411    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-host\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.712588 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712455    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-cnibin\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.712588 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712488    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-socket-dir\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.712588 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712536    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-sys\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.712588 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712566    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-var-lib-cni-multus\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.712789 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712598    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-var-lib-kubelet\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.712789 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712628    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-hostroot\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.712789 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712655    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-etc-kubernetes\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.712789 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712682    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-registration-dir\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.712789 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712715    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-etc-selinux\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.712789 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712739    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-var-lib-kubelet\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.712789 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712763    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txj26\" (UniqueName: \"kubernetes.io/projected/914c4cb9-1867-478b-8e4e-39185d5fc46d-kube-api-access-txj26\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.713267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712822    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e24f48e4-042d-469d-a94a-7bb0451cebf7-cni-binary-copy\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.713267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712867    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-run-netns\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.713267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712912    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-run-multus-certs\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.713267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712960    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39947d46-b67f-4338-b764-431cbb707b86-host-slash\") pod \"iptables-alerter-qpgmv\" (UID: \"39947d46-b67f-4338-b764-431cbb707b86\") " pod="openshift-network-operator/iptables-alerter-qpgmv"
Sep 09 10:26:02.713267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.712988    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9ll7\" (UniqueName: \"kubernetes.io/projected/39947d46-b67f-4338-b764-431cbb707b86-kube-api-access-t9ll7\") pod \"iptables-alerter-qpgmv\" (UID: \"39947d46-b67f-4338-b764-431cbb707b86\") " pod="openshift-network-operator/iptables-alerter-qpgmv"
Sep 09 10:26:02.713267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713014    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-sysconfig\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.713267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713040    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-sysctl-conf\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.713267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713065    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-tuned\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.713267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713105    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-898jz\" (UniqueName: \"kubernetes.io/projected/e24f48e4-042d-469d-a94a-7bb0451cebf7-kube-api-access-898jz\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.713267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713151    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-multus-cni-dir\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.713267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713182    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-multus-socket-dir-parent\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.713267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713207    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-device-dir\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.713267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-sysctl-d\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713277    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-systemd\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713307    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-lib-modules\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713332    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-system-cni-dir\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713356    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-multus-conf-dir\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713395    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-kubernetes\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713415    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-run\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713430    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/914c4cb9-1867-478b-8e4e-39185d5fc46d-tmp\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713468    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e24f48e4-042d-469d-a94a-7bb0451cebf7-multus-daemon-config\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713504    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713533    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-sys-fs\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713552    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713561    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q4tn\" (UniqueName: \"kubernetes.io/projected/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-kube-api-access-5q4tn\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713591    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39947d46-b67f-4338-b764-431cbb707b86-iptables-alerter-script\") pod \"iptables-alerter-qpgmv\" (UID: \"39947d46-b67f-4338-b764-431cbb707b86\") " pod="openshift-network-operator/iptables-alerter-qpgmv"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713617    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-run-k8s-cni-cncf-io\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.713900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.713642    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-var-lib-cni-bin\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.715884 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.715798    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:02.716003 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:02.715876    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:02.718703 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.718687    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:02.718782 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:02.718742    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:02.720806 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.720762    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.723292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.723276    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dh892"
Sep 09 10:26:02.725695 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.725677    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:02.725815 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:02.725729    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:02.735109 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.735047    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 10:26:02.735109 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.735054    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 10:26:02.735257 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.735117    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 10:26:02.735257 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.735076    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 10:26:02.735659 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.735081    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 10:26:02.735659 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.735130    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 10:26:02.735818 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.735701    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 10:26:02.736506 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.736485    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 10:26:02.738206 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.736681    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t9q8x\""
Sep 09 10:26:02.738206 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.736742    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bshz9\""
Sep 09 10:26:02.738206 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.736785    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-72tcv\""
Sep 09 10:26:02.738206 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.737095    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vk6hs\""
Sep 09 10:26:02.738871 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.738834    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 10:26:02.741955 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.741934    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 10:26:02.743235 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.743210    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 10:26:02.743318 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.743255    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 10:26:02.744029 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.743849    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 10:26:02.744029 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.743879    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 10:26:02.744029 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.743889    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9bgk6\""
Sep 09 10:26:02.744029 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.743922    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rvg4l\""
Sep 09 10:26:02.744029 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.743926    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 10:26:02.744277 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.744264    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 10:26:02.746836 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.746814    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 10:26:02.747887 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.747870    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 10:26:02.767883 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.767860    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 10:26:02.789666 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.789642    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4mbcv"
Sep 09 10:26:02.805248 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.805225    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 10:26:02.808906 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.808888    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4mbcv"
Sep 09 10:26:02.814389 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814371    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:02.814462 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814401    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55711478-e733-4bdb-a947-0b6f37cc3f75-serviceca\") pod \"node-ca-5j47k\" (UID: \"55711478-e733-4bdb-a947-0b6f37cc3f75\") " pod="openshift-image-registry/node-ca-5j47k"
Sep 09 10:26:02.814462 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814441    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4mh\" (UniqueName: \"kubernetes.io/projected/55711478-e733-4bdb-a947-0b6f37cc3f75-kube-api-access-dc4mh\") pod \"node-ca-5j47k\" (UID: \"55711478-e733-4bdb-a947-0b6f37cc3f75\") " pod="openshift-image-registry/node-ca-5j47k"
Sep 09 10:26:02.814554 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814491    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-run-netns\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.814554 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814517    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8b797865-8296-4715-b5cb-69bb3f884b62-ovnkube-script-lib\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.814554 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814542    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-kubernetes\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.814737 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814569    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-run\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.814737 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814617    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-kubernetes\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.814737 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814664    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/914c4cb9-1867-478b-8e4e-39185d5fc46d-tmp\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.814737 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814704    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-run\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.814737 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814719    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q4tn\" (UniqueName: \"kubernetes.io/projected/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-kube-api-access-5q4tn\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.814957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814755    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39947d46-b67f-4338-b764-431cbb707b86-iptables-alerter-script\") pod \"iptables-alerter-qpgmv\" (UID: \"39947d46-b67f-4338-b764-431cbb707b86\") " pod="openshift-network-operator/iptables-alerter-qpgmv"
Sep 09 10:26:02.814957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814799    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8b797865-8296-4715-b5cb-69bb3f884b62-ovnkube-config\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.814957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814824    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cll7w\" (UniqueName: \"kubernetes.io/projected/8b797865-8296-4715-b5cb-69bb3f884b62-kube-api-access-cll7w\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.814957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814880    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/935d141b-8ba9-42a3-9a46-91f75e24a0bf-hosts-file\") pod \"node-resolver-dh892\" (UID: \"935d141b-8ba9-42a3-9a46-91f75e24a0bf\") " pod="openshift-dns/node-resolver-dh892"
Sep 09 10:26:02.814957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814927    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-var-lib-openvswitch\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.815113 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.814984    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8b797865-8296-4715-b5cb-69bb3f884b62-env-overrides\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.815113 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815012    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/935d141b-8ba9-42a3-9a46-91f75e24a0bf-tmp-dir\") pod \"node-resolver-dh892\" (UID: \"935d141b-8ba9-42a3-9a46-91f75e24a0bf\") " pod="openshift-dns/node-resolver-dh892"
Sep 09 10:26:02.815113 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815032    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-cnibin\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815113 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815050    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq2l8\" (UniqueName: \"kubernetes.io/projected/30ae792d-a70e-4cf5-8d08-cee325dbd625-kube-api-access-bq2l8\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.815113 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815066    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-systemd-units\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.815113 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815084    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-var-lib-kubelet\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815131    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-cnibin\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815137    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-hostroot\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815164    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-etc-kubernetes\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815168    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-var-lib-kubelet\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815181    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-hostroot\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815188    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39947d46-b67f-4338-b764-431cbb707b86-host-slash\") pod \"iptables-alerter-qpgmv\" (UID: \"39947d46-b67f-4338-b764-431cbb707b86\") " pod="openshift-network-operator/iptables-alerter-qpgmv"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815224    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39947d46-b67f-4338-b764-431cbb707b86-host-slash\") pod \"iptables-alerter-qpgmv\" (UID: \"39947d46-b67f-4338-b764-431cbb707b86\") " pod="openshift-network-operator/iptables-alerter-qpgmv"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815228    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-kubelet-config\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815250    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-etc-kubernetes\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815258    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq5gp\" (UniqueName: \"kubernetes.io/projected/935d141b-8ba9-42a3-9a46-91f75e24a0bf-kube-api-access-mq5gp\") pod \"node-resolver-dh892\" (UID: \"935d141b-8ba9-42a3-9a46-91f75e24a0bf\") " pod="openshift-dns/node-resolver-dh892"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815292    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55711478-e733-4bdb-a947-0b6f37cc3f75-host\") pod \"node-ca-5j47k\" (UID: \"55711478-e733-4bdb-a947-0b6f37cc3f75\") " pod="openshift-image-registry/node-ca-5j47k"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815294    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39947d46-b67f-4338-b764-431cbb707b86-iptables-alerter-script\") pod \"iptables-alerter-qpgmv\" (UID: \"39947d46-b67f-4338-b764-431cbb707b86\") " pod="openshift-network-operator/iptables-alerter-qpgmv"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815326    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txj26\" (UniqueName: \"kubernetes.io/projected/914c4cb9-1867-478b-8e4e-39185d5fc46d-kube-api-access-txj26\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.815395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815356    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-run-netns\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815432    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-run-netns\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815432    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-run-multus-certs\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815464    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-cni-netd\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815489    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-run-multus-certs\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815492    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815521    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-sysconfig\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815554    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-sysctl-conf\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815577    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-sysconfig\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815579    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815582    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-898jz\" (UniqueName: \"kubernetes.io/projected/e24f48e4-042d-469d-a94a-7bb0451cebf7-kube-api-access-898jz\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815643    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-etc-openvswitch\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815666    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-cni-bin\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815689    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-multus-socket-dir-parent\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815708    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-sysctl-conf\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815721    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ae792d-a70e-4cf5-8d08-cee325dbd625-system-cni-dir\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815754    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-node-log\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.815974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815801    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-multus-socket-dir-parent\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815816    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-systemd\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815840    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-multus-conf-dir\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815858    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30ae792d-a70e-4cf5-8d08-cee325dbd625-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815887    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815900    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-systemd\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815914    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e24f48e4-042d-469d-a94a-7bb0451cebf7-multus-daemon-config\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815932    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-multus-conf-dir\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815958    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815976    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-sys-fs\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.815998    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-run-k8s-cni-cncf-io\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816024    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-var-lib-cni-bin\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816037    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816054    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9fd12634-040b-49e0-b16b-30c8ac321311-agent-certs\") pod \"konnectivity-agent-4t6qk\" (UID: \"9fd12634-040b-49e0-b16b-30c8ac321311\") " pod="kube-system/konnectivity-agent-4t6qk"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816064    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-sys-fs\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816056    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-run-k8s-cni-cncf-io\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816081    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-modprobe-d\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.816605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816087    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-var-lib-cni-bin\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816114    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-host\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816147    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-socket-dir\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816169    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-dbus\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816201    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-modprobe-d\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816206    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-host\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816229    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30ae792d-a70e-4cf5-8d08-cee325dbd625-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816258    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-socket-dir\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816261    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9fd12634-040b-49e0-b16b-30c8ac321311-konnectivity-ca\") pod \"konnectivity-agent-4t6qk\" (UID: \"9fd12634-040b-49e0-b16b-30c8ac321311\") " pod="kube-system/konnectivity-agent-4t6qk"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816307    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm9rp\" (UniqueName: \"kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp\") pod \"network-check-target-rsgg6\" (UID: \"0a17e4ac-d603-4db3-9d72-578389bf0db5\") " pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816338    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-sys\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816381    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-var-lib-cni-multus\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816407    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-registration-dir\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816432    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-sys\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816449    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-host-var-lib-cni-multus\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816433    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-etc-selinux\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816434    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e24f48e4-042d-469d-a94a-7bb0451cebf7-multus-daemon-config\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.817388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816478    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-etc-selinux\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816487    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-run-systemd\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816491    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-registration-dir\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816519    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-run-openvswitch\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816536    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-run-ovn\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816553    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-var-lib-kubelet\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816576    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e24f48e4-042d-469d-a94a-7bb0451cebf7-cni-binary-copy\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816603    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9ll7\" (UniqueName: \"kubernetes.io/projected/39947d46-b67f-4338-b764-431cbb707b86-kube-api-access-t9ll7\") pod \"iptables-alerter-qpgmv\" (UID: \"39947d46-b67f-4338-b764-431cbb707b86\") " pod="openshift-network-operator/iptables-alerter-qpgmv"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816619    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-var-lib-kubelet\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816629    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-log-socket\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816652    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-run-ovn-kubernetes\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816668    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b797865-8296-4715-b5cb-69bb3f884b62-ovn-node-metrics-cert\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816685    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-tuned\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816706    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30ae792d-a70e-4cf5-8d08-cee325dbd625-cnibin\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816751    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-multus-cni-dir\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816766    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-device-dir\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816785    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30ae792d-a70e-4cf5-8d08-cee325dbd625-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.818012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816827    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30ae792d-a70e-4cf5-8d08-cee325dbd625-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.818551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816844    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-device-dir\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.818551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816847    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-multus-cni-dir\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.818551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816855    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9jbf\" (UniqueName: \"kubernetes.io/projected/6969769c-d30e-4cbb-afc5-9df38c741545-kube-api-access-f9jbf\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:02.818551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816887    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-kubelet\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.818551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816908    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-slash\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.818551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816923    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-sysctl-d\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.818551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.816943    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-lib-modules\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.818551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.817015    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e24f48e4-042d-469d-a94a-7bb0451cebf7-cni-binary-copy\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.818551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.817025    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-lib-modules\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.818551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.817058    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-sysctl-d\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.818551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.817091    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-system-cni-dir\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.818551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.817230    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e24f48e4-042d-469d-a94a-7bb0451cebf7-system-cni-dir\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.818920 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.818803    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/914c4cb9-1867-478b-8e4e-39185d5fc46d-tmp\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.818920 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.818913    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/914c4cb9-1867-478b-8e4e-39185d5fc46d-etc-tuned\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.832786 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.832764    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q4tn\" (UniqueName: \"kubernetes.io/projected/ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59-kube-api-access-5q4tn\") pod \"aws-ebs-csi-driver-node-q6pwn\" (UID: \"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:02.837470 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.837424    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal" event={"ID":"9a4115943f85345d07176841ce75f166","Type":"ContainerStarted","Data":"f8a1af195bd33f42c294742bc9f30858cd1f0d08ea350eaf84e660f537a993c5"}
Sep 09 10:26:02.838496 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.838461    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txj26\" (UniqueName: \"kubernetes.io/projected/914c4cb9-1867-478b-8e4e-39185d5fc46d-kube-api-access-txj26\") pod \"tuned-sh98r\" (UID: \"914c4cb9-1867-478b-8e4e-39185d5fc46d\") " pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:02.838679 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.838660    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-193.ec2.internal" event={"ID":"046071a0f74eb379e298ca5ffe8a80f1","Type":"ContainerStarted","Data":"a6a8f8e9cb2ba9f9ed6633adad8d8f328c8c237c27590fb63254f0250f3d1402"}
Sep 09 10:26:02.840023 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.840004    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-898jz\" (UniqueName: \"kubernetes.io/projected/e24f48e4-042d-469d-a94a-7bb0451cebf7-kube-api-access-898jz\") pod \"multus-4qlxw\" (UID: \"e24f48e4-042d-469d-a94a-7bb0451cebf7\") " pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:02.842184 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.842167    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9ll7\" (UniqueName: \"kubernetes.io/projected/39947d46-b67f-4338-b764-431cbb707b86-kube-api-access-t9ll7\") pod \"iptables-alerter-qpgmv\" (UID: \"39947d46-b67f-4338-b764-431cbb707b86\") " pod="openshift-network-operator/iptables-alerter-qpgmv"
Sep 09 10:26:02.917704 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.917669    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9fd12634-040b-49e0-b16b-30c8ac321311-konnectivity-ca\") pod \"konnectivity-agent-4t6qk\" (UID: \"9fd12634-040b-49e0-b16b-30c8ac321311\") " pod="kube-system/konnectivity-agent-4t6qk"
Sep 09 10:26:02.917877 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.917707    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9rp\" (UniqueName: \"kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp\") pod \"network-check-target-rsgg6\" (UID: \"0a17e4ac-d603-4db3-9d72-578389bf0db5\") " pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:02.917877 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.917740    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-run-systemd\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.917877 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.917759    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-run-openvswitch\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.917877 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.917817    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-run-systemd\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.917877 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.917824    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-run-openvswitch\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.918072 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.917943    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-run-ovn\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.918072 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.917980    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-log-socket\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.918072 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918021    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-run-ovn-kubernetes\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.918072 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918048    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-log-socket\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.918072 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918051    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b797865-8296-4715-b5cb-69bb3f884b62-ovn-node-metrics-cert\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918092    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30ae792d-a70e-4cf5-8d08-cee325dbd625-cnibin\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918053    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-run-ovn-kubernetes\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918112    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30ae792d-a70e-4cf5-8d08-cee325dbd625-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918022    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-run-ovn\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918130    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30ae792d-a70e-4cf5-8d08-cee325dbd625-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918147    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30ae792d-a70e-4cf5-8d08-cee325dbd625-cnibin\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918156    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9jbf\" (UniqueName: \"kubernetes.io/projected/6969769c-d30e-4cbb-afc5-9df38c741545-kube-api-access-f9jbf\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918181    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-kubelet\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918198    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-slash\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918215    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918234    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55711478-e733-4bdb-a947-0b6f37cc3f75-serviceca\") pod \"node-ca-5j47k\" (UID: \"55711478-e733-4bdb-a947-0b6f37cc3f75\") " pod="openshift-image-registry/node-ca-5j47k"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918258    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4mh\" (UniqueName: \"kubernetes.io/projected/55711478-e733-4bdb-a947-0b6f37cc3f75-kube-api-access-dc4mh\") pod \"node-ca-5j47k\" (UID: \"55711478-e733-4bdb-a947-0b6f37cc3f75\") " pod="openshift-image-registry/node-ca-5j47k"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918282    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-run-netns\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918284    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-kubelet\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918321    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8b797865-8296-4715-b5cb-69bb3f884b62-ovnkube-script-lib\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.918336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918324    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-run-netns\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918356    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8b797865-8296-4715-b5cb-69bb3f884b62-ovnkube-config\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918412    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cll7w\" (UniqueName: \"kubernetes.io/projected/8b797865-8296-4715-b5cb-69bb3f884b62-kube-api-access-cll7w\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918438    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/935d141b-8ba9-42a3-9a46-91f75e24a0bf-hosts-file\") pod \"node-resolver-dh892\" (UID: \"935d141b-8ba9-42a3-9a46-91f75e24a0bf\") " pod="openshift-dns/node-resolver-dh892"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918464    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-var-lib-openvswitch\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918486    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9fd12634-040b-49e0-b16b-30c8ac321311-konnectivity-ca\") pod \"konnectivity-agent-4t6qk\" (UID: \"9fd12634-040b-49e0-b16b-30c8ac321311\") " pod="kube-system/konnectivity-agent-4t6qk"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:02.918629    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918493    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8b797865-8296-4715-b5cb-69bb3f884b62-env-overrides\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918715    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30ae792d-a70e-4cf5-8d08-cee325dbd625-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918741    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/935d141b-8ba9-42a3-9a46-91f75e24a0bf-tmp-dir\") pod \"node-resolver-dh892\" (UID: \"935d141b-8ba9-42a3-9a46-91f75e24a0bf\") " pod="openshift-dns/node-resolver-dh892"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918778    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq2l8\" (UniqueName: \"kubernetes.io/projected/30ae792d-a70e-4cf5-8d08-cee325dbd625-kube-api-access-bq2l8\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918803    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-systemd-units\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918840    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-kubelet-config\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:02.918863    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret podName:11460f81-a3fc-4a03-aa3f-79dc61b95a4a nodeName:}" failed. No retries permitted until 2026-09-09 10:26:03.418840006 +0000 UTC m=+3.082478975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret") pod "global-pull-secret-syncer-crdlr" (UID: "11460f81-a3fc-4a03-aa3f-79dc61b95a4a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918869    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55711478-e733-4bdb-a947-0b6f37cc3f75-serviceca\") pod \"node-ca-5j47k\" (UID: \"55711478-e733-4bdb-a947-0b6f37cc3f75\") " pod="openshift-image-registry/node-ca-5j47k"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918902    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq5gp\" (UniqueName: \"kubernetes.io/projected/935d141b-8ba9-42a3-9a46-91f75e24a0bf-kube-api-access-mq5gp\") pod \"node-resolver-dh892\" (UID: \"935d141b-8ba9-42a3-9a46-91f75e24a0bf\") " pod="openshift-dns/node-resolver-dh892"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918943    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55711478-e733-4bdb-a947-0b6f37cc3f75-host\") pod \"node-ca-5j47k\" (UID: \"55711478-e733-4bdb-a947-0b6f37cc3f75\") " pod="openshift-image-registry/node-ca-5j47k"
Sep 09 10:26:02.919110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918956    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/935d141b-8ba9-42a3-9a46-91f75e24a0bf-hosts-file\") pod \"node-resolver-dh892\" (UID: \"935d141b-8ba9-42a3-9a46-91f75e24a0bf\") " pod="openshift-dns/node-resolver-dh892"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918978    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-cni-netd\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919015    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919047    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-etc-openvswitch\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919078    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8b797865-8296-4715-b5cb-69bb3f884b62-ovnkube-script-lib\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919113    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-systemd-units\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919132    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-cni-bin\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918913    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-kubelet-config\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.918385    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-slash\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919205    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-var-lib-openvswitch\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919081    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-cni-bin\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919245    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ae792d-a70e-4cf5-8d08-cee325dbd625-system-cni-dir\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919273    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-node-log\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919289    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30ae792d-a70e-4cf5-8d08-cee325dbd625-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919302    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30ae792d-a70e-4cf5-8d08-cee325dbd625-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919329    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919335    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8b797865-8296-4715-b5cb-69bb3f884b62-env-overrides\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.919947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919353    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-node-log\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919370    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9fd12634-040b-49e0-b16b-30c8ac321311-agent-certs\") pod \"konnectivity-agent-4t6qk\" (UID: \"9fd12634-040b-49e0-b16b-30c8ac321311\") " pod="kube-system/konnectivity-agent-4t6qk"
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919400    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-dbus\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919423    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/935d141b-8ba9-42a3-9a46-91f75e24a0bf-tmp-dir\") pod \"node-resolver-dh892\" (UID: \"935d141b-8ba9-42a3-9a46-91f75e24a0bf\") " pod="openshift-dns/node-resolver-dh892"
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919433    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-cni-netd\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919426    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30ae792d-a70e-4cf5-8d08-cee325dbd625-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919472    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55711478-e733-4bdb-a947-0b6f37cc3f75-host\") pod \"node-ca-5j47k\" (UID: \"55711478-e733-4bdb-a947-0b6f37cc3f75\") " pod="openshift-image-registry/node-ca-5j47k"
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919508    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919515    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30ae792d-a70e-4cf5-8d08-cee325dbd625-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919399    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ae792d-a70e-4cf5-8d08-cee325dbd625-system-cni-dir\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919517    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b797865-8296-4715-b5cb-69bb3f884b62-etc-openvswitch\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919563    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-dbus\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:02.919594    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:02.919646    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs podName:6969769c-d30e-4cbb-afc5-9df38c741545 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:03.419630123 +0000 UTC m=+3.083269098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs") pod "network-metrics-daemon-llgq5" (UID: "6969769c-d30e-4cbb-afc5-9df38c741545") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919697    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8b797865-8296-4715-b5cb-69bb3f884b62-ovnkube-config\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.920615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.919812    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30ae792d-a70e-4cf5-8d08-cee325dbd625-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.921670 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.921616    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b797865-8296-4715-b5cb-69bb3f884b62-ovn-node-metrics-cert\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:02.923020 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.922990    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9fd12634-040b-49e0-b16b-30c8ac321311-agent-certs\") pod \"konnectivity-agent-4t6qk\" (UID: \"9fd12634-040b-49e0-b16b-30c8ac321311\") " pod="kube-system/konnectivity-agent-4t6qk"
Sep 09 10:26:02.926606 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:02.926589    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 10:26:02.926606 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:02.926608    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 10:26:02.926718 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:02.926619    2570 projected.go:196] Error preparing data for projected volume kube-api-access-tm9rp for pod openshift-network-diagnostics/network-check-target-rsgg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:02.926718 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:02.926674    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp podName:0a17e4ac-d603-4db3-9d72-578389bf0db5 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:03.426661265 +0000 UTC m=+3.090300244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tm9rp" (UniqueName: "kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp") pod "network-check-target-rsgg6" (UID: "0a17e4ac-d603-4db3-9d72-578389bf0db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:02.929587 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.929563    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4mh\" (UniqueName: \"kubernetes.io/projected/55711478-e733-4bdb-a947-0b6f37cc3f75-kube-api-access-dc4mh\") pod \"node-ca-5j47k\" (UID: \"55711478-e733-4bdb-a947-0b6f37cc3f75\") " pod="openshift-image-registry/node-ca-5j47k"
Sep 09 10:26:02.932498 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.932448    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq2l8\" (UniqueName: \"kubernetes.io/projected/30ae792d-a70e-4cf5-8d08-cee325dbd625-kube-api-access-bq2l8\") pod \"multus-additional-cni-plugins-vlj8h\" (UID: \"30ae792d-a70e-4cf5-8d08-cee325dbd625\") " pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:02.932574 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.932548    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9jbf\" (UniqueName: \"kubernetes.io/projected/6969769c-d30e-4cbb-afc5-9df38c741545-kube-api-access-f9jbf\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:02.932628 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.932609    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq5gp\" (UniqueName: \"kubernetes.io/projected/935d141b-8ba9-42a3-9a46-91f75e24a0bf-kube-api-access-mq5gp\") pod \"node-resolver-dh892\" (UID: \"935d141b-8ba9-42a3-9a46-91f75e24a0bf\") " pod="openshift-dns/node-resolver-dh892"
Sep 09 10:26:02.932706 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:02.932690    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cll7w\" (UniqueName: \"kubernetes.io/projected/8b797865-8296-4715-b5cb-69bb3f884b62-kube-api-access-cll7w\") pod \"ovnkube-node-qs8qn\" (UID: \"8b797865-8296-4715-b5cb-69bb3f884b62\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:03.010190 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.010159    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sh98r"
Sep 09 10:26:03.016368 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.016336    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4qlxw"
Sep 09 10:26:03.021157 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:03.021128    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914c4cb9_1867_478b_8e4e_39185d5fc46d.slice/crio-ecbf49064a3df155a99168047414eacdb00a2b388aa447fc917fd1f59c7af58f WatchSource:0}: Error finding container ecbf49064a3df155a99168047414eacdb00a2b388aa447fc917fd1f59c7af58f: Status 404 returned error can't find the container with id ecbf49064a3df155a99168047414eacdb00a2b388aa447fc917fd1f59c7af58f
Sep 09 10:26:03.024612 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.024577    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn"
Sep 09 10:26:03.030509 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.029807    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qpgmv"
Sep 09 10:26:03.030509 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:03.030170    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24f48e4_042d_469d_a94a_7bb0451cebf7.slice/crio-efc1707c43df7dec8072e7def11324adf5c781937939c050c49b0bdfa660166e WatchSource:0}: Error finding container efc1707c43df7dec8072e7def11324adf5c781937939c050c49b0bdfa660166e: Status 404 returned error can't find the container with id efc1707c43df7dec8072e7def11324adf5c781937939c050c49b0bdfa660166e
Sep 09 10:26:03.035735 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.035712    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5j47k"
Sep 09 10:26:03.037300 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:03.037273    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7dfdb6_b2bd_4d77_94fa_b6ee57306d59.slice/crio-733fe987495b2241783e0001f55c22462c3cf95120d22aba0bee53275dffa504 WatchSource:0}: Error finding container 733fe987495b2241783e0001f55c22462c3cf95120d22aba0bee53275dffa504: Status 404 returned error can't find the container with id 733fe987495b2241783e0001f55c22462c3cf95120d22aba0bee53275dffa504
Sep 09 10:26:03.041533 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:03.041507    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39947d46_b67f_4338_b764_431cbb707b86.slice/crio-d9b785ae543e35c35c14afb46bba845adf75d3dc97149a1ebd998e39e93211f8 WatchSource:0}: Error finding container d9b785ae543e35c35c14afb46bba845adf75d3dc97149a1ebd998e39e93211f8: Status 404 returned error can't find the container with id d9b785ae543e35c35c14afb46bba845adf75d3dc97149a1ebd998e39e93211f8
Sep 09 10:26:03.043078 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.042827    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4t6qk"
Sep 09 10:26:03.046610 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:03.046582    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55711478_e733_4bdb_a947_0b6f37cc3f75.slice/crio-2075033b071dc06a45ab628516329126f5df1ab1843f15c1f9b681525669a0ec WatchSource:0}: Error finding container 2075033b071dc06a45ab628516329126f5df1ab1843f15c1f9b681525669a0ec: Status 404 returned error can't find the container with id 2075033b071dc06a45ab628516329126f5df1ab1843f15c1f9b681525669a0ec
Sep 09 10:26:03.049362 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.049342    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:03.054757 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.054582    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vlj8h"
Sep 09 10:26:03.059179 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.059159    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dh892"
Sep 09 10:26:03.061854 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:03.061771    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b797865_8296_4715_b5cb_69bb3f884b62.slice/crio-06cf553946760471be781b873213eb941313e2e6b150fbf127cf880bad878f31 WatchSource:0}: Error finding container 06cf553946760471be781b873213eb941313e2e6b150fbf127cf880bad878f31: Status 404 returned error can't find the container with id 06cf553946760471be781b873213eb941313e2e6b150fbf127cf880bad878f31
Sep 09 10:26:03.065744 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:03.065720    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ae792d_a70e_4cf5_8d08_cee325dbd625.slice/crio-5b67070b6ccc1460a5b13e1e111ac4725d3e3fae7b1ae9604f522d6f5a16a810 WatchSource:0}: Error finding container 5b67070b6ccc1460a5b13e1e111ac4725d3e3fae7b1ae9604f522d6f5a16a810: Status 404 returned error can't find the container with id 5b67070b6ccc1460a5b13e1e111ac4725d3e3fae7b1ae9604f522d6f5a16a810
Sep 09 10:26:03.070526 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:03.070501    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935d141b_8ba9_42a3_9a46_91f75e24a0bf.slice/crio-ecedab41998be01694a34e3363eceb378c928011c7f896aca9920ff94069f3ff WatchSource:0}: Error finding container ecedab41998be01694a34e3363eceb378c928011c7f896aca9920ff94069f3ff: Status 404 returned error can't find the container with id ecedab41998be01694a34e3363eceb378c928011c7f896aca9920ff94069f3ff
Sep 09 10:26:03.098342 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.098315    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 10:26:03.424671 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.423839    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:03.424671 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.423937    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:03.424671 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:03.424074    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:03.424671 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:03.424141    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs podName:6969769c-d30e-4cbb-afc5-9df38c741545 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:04.424119574 +0000 UTC m=+4.087758546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs") pod "network-metrics-daemon-llgq5" (UID: "6969769c-d30e-4cbb-afc5-9df38c741545") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:03.424671 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:03.424196    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:03.424671 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:03.424248    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret podName:11460f81-a3fc-4a03-aa3f-79dc61b95a4a nodeName:}" failed. No retries permitted until 2026-09-09 10:26:04.424230332 +0000 UTC m=+4.087869295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret") pod "global-pull-secret-syncer-crdlr" (UID: "11460f81-a3fc-4a03-aa3f-79dc61b95a4a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:03.525597 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.524836    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9rp\" (UniqueName: \"kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp\") pod \"network-check-target-rsgg6\" (UID: \"0a17e4ac-d603-4db3-9d72-578389bf0db5\") " pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:03.525597 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:03.525064    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 10:26:03.525597 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:03.525087    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 10:26:03.525597 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:03.525101    2570 projected.go:196] Error preparing data for projected volume kube-api-access-tm9rp for pod openshift-network-diagnostics/network-check-target-rsgg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:03.525597 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:03.525162    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp podName:0a17e4ac-d603-4db3-9d72-578389bf0db5 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:04.52514297 +0000 UTC m=+4.188781946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm9rp" (UniqueName: "kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp") pod "network-check-target-rsgg6" (UID: "0a17e4ac-d603-4db3-9d72-578389bf0db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:03.812216 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.812105    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 10:21:03 +0000 UTC" deadline="2028-06-07 01:10:53.134973632 +0000 UTC"
Sep 09 10:26:03.812216 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.812157    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15278h44m49.322821931s"
Sep 09 10:26:03.834493 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.834048    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:03.834493 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:03.834173    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:03.843416 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.843369    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn" event={"ID":"8b797865-8296-4715-b5cb-69bb3f884b62","Type":"ContainerStarted","Data":"06cf553946760471be781b873213eb941313e2e6b150fbf127cf880bad878f31"}
Sep 09 10:26:03.844930 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.844842    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn" event={"ID":"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59","Type":"ContainerStarted","Data":"733fe987495b2241783e0001f55c22462c3cf95120d22aba0bee53275dffa504"}
Sep 09 10:26:03.847374 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.847321    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4qlxw" event={"ID":"e24f48e4-042d-469d-a94a-7bb0451cebf7","Type":"ContainerStarted","Data":"efc1707c43df7dec8072e7def11324adf5c781937939c050c49b0bdfa660166e"}
Sep 09 10:26:03.849042 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.849019    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlj8h" event={"ID":"30ae792d-a70e-4cf5-8d08-cee325dbd625","Type":"ContainerStarted","Data":"5b67070b6ccc1460a5b13e1e111ac4725d3e3fae7b1ae9604f522d6f5a16a810"}
Sep 09 10:26:03.850623 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.850598    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4t6qk" event={"ID":"9fd12634-040b-49e0-b16b-30c8ac321311","Type":"ContainerStarted","Data":"0f9001288260958ee55c19ff24fb844f023de11be3ac3531d668d4eadc8f9633"}
Sep 09 10:26:03.852123 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.852090    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5j47k" event={"ID":"55711478-e733-4bdb-a947-0b6f37cc3f75","Type":"ContainerStarted","Data":"2075033b071dc06a45ab628516329126f5df1ab1843f15c1f9b681525669a0ec"}
Sep 09 10:26:03.853669 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.853644    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qpgmv" event={"ID":"39947d46-b67f-4338-b764-431cbb707b86","Type":"ContainerStarted","Data":"d9b785ae543e35c35c14afb46bba845adf75d3dc97149a1ebd998e39e93211f8"}
Sep 09 10:26:03.856172 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.856115    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sh98r" event={"ID":"914c4cb9-1867-478b-8e4e-39185d5fc46d","Type":"ContainerStarted","Data":"ecbf49064a3df155a99168047414eacdb00a2b388aa447fc917fd1f59c7af58f"}
Sep 09 10:26:03.859202 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:03.859138    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dh892" event={"ID":"935d141b-8ba9-42a3-9a46-91f75e24a0bf","Type":"ContainerStarted","Data":"ecedab41998be01694a34e3363eceb378c928011c7f896aca9920ff94069f3ff"}
Sep 09 10:26:04.439885 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:04.433517    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:04.439885 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:04.433626    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:04.441908 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:04.440343    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:04.441908 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:04.440449    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs podName:6969769c-d30e-4cbb-afc5-9df38c741545 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:06.440424419 +0000 UTC m=+6.104063390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs") pod "network-metrics-daemon-llgq5" (UID: "6969769c-d30e-4cbb-afc5-9df38c741545") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:04.441908 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:04.440561    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:04.441908 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:04.440665    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret podName:11460f81-a3fc-4a03-aa3f-79dc61b95a4a nodeName:}" failed. No retries permitted until 2026-09-09 10:26:06.440645248 +0000 UTC m=+6.104284221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret") pod "global-pull-secret-syncer-crdlr" (UID: "11460f81-a3fc-4a03-aa3f-79dc61b95a4a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:04.535126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:04.535044    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9rp\" (UniqueName: \"kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp\") pod \"network-check-target-rsgg6\" (UID: \"0a17e4ac-d603-4db3-9d72-578389bf0db5\") " pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:04.535326 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:04.535204    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 10:26:04.535326 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:04.535224    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 10:26:04.535326 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:04.535237    2570 projected.go:196] Error preparing data for projected volume kube-api-access-tm9rp for pod openshift-network-diagnostics/network-check-target-rsgg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:04.535501 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:04.535358    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp podName:0a17e4ac-d603-4db3-9d72-578389bf0db5 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:06.535335166 +0000 UTC m=+6.198974141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm9rp" (UniqueName: "kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp") pod "network-check-target-rsgg6" (UID: "0a17e4ac-d603-4db3-9d72-578389bf0db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:04.813447 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:04.813335    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 10:21:03 +0000 UTC" deadline="2028-05-12 05:15:02.652019632 +0000 UTC"
Sep 09 10:26:04.813447 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:04.813403    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14658h48m57.838621155s"
Sep 09 10:26:04.835663 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:04.835632    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:04.835837 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:04.835768    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:04.835901 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:04.835868    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:04.836013 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:04.835960    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:05.833682 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:05.833643    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:05.834179 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:05.833827    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:06.453603 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:06.452772    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:06.453603 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:06.452872    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:06.453603 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:06.453000    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:06.453603 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:06.453067    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs podName:6969769c-d30e-4cbb-afc5-9df38c741545 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:10.453046664 +0000 UTC m=+10.116685620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs") pod "network-metrics-daemon-llgq5" (UID: "6969769c-d30e-4cbb-afc5-9df38c741545") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:06.453603 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:06.453504    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:06.453603 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:06.453565    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret podName:11460f81-a3fc-4a03-aa3f-79dc61b95a4a nodeName:}" failed. No retries permitted until 2026-09-09 10:26:10.45354706 +0000 UTC m=+10.117186029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret") pod "global-pull-secret-syncer-crdlr" (UID: "11460f81-a3fc-4a03-aa3f-79dc61b95a4a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:06.553931 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:06.553892    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9rp\" (UniqueName: \"kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp\") pod \"network-check-target-rsgg6\" (UID: \"0a17e4ac-d603-4db3-9d72-578389bf0db5\") " pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:06.554194 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:06.554113    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 10:26:06.554194 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:06.554137    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 10:26:06.554194 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:06.554151    2570 projected.go:196] Error preparing data for projected volume kube-api-access-tm9rp for pod openshift-network-diagnostics/network-check-target-rsgg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:06.554376 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:06.554214    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp podName:0a17e4ac-d603-4db3-9d72-578389bf0db5 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:10.554193478 +0000 UTC m=+10.217832435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm9rp" (UniqueName: "kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp") pod "network-check-target-rsgg6" (UID: "0a17e4ac-d603-4db3-9d72-578389bf0db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:06.833736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:06.833230    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:06.833736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:06.833268    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:06.833736 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:06.833385    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:06.833736 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:06.833519    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:07.834087 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:07.834046    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:07.834581 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:07.834187    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:08.835004 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:08.834968    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:08.835004 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:08.835003    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:08.835576 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:08.835142    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:08.835576 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:08.835327    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:09.834473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:09.833900    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:09.834473 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:09.834051    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:10.490421 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:10.490240    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:10.490421 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:10.490337    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:10.490950 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:10.490444    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:10.490950 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:10.490546    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret podName:11460f81-a3fc-4a03-aa3f-79dc61b95a4a nodeName:}" failed. No retries permitted until 2026-09-09 10:26:18.490519808 +0000 UTC m=+18.154158782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret") pod "global-pull-secret-syncer-crdlr" (UID: "11460f81-a3fc-4a03-aa3f-79dc61b95a4a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:10.490950 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:10.490565    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:10.490950 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:10.490633    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs podName:6969769c-d30e-4cbb-afc5-9df38c741545 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:18.490616155 +0000 UTC m=+18.154255111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs") pod "network-metrics-daemon-llgq5" (UID: "6969769c-d30e-4cbb-afc5-9df38c741545") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:10.591208 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:10.591165    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9rp\" (UniqueName: \"kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp\") pod \"network-check-target-rsgg6\" (UID: \"0a17e4ac-d603-4db3-9d72-578389bf0db5\") " pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:10.591406 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:10.591387    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 10:26:10.591474 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:10.591408    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 10:26:10.591474 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:10.591422    2570 projected.go:196] Error preparing data for projected volume kube-api-access-tm9rp for pod openshift-network-diagnostics/network-check-target-rsgg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:10.591580 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:10.591483    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp podName:0a17e4ac-d603-4db3-9d72-578389bf0db5 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:18.591463821 +0000 UTC m=+18.255102789 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm9rp" (UniqueName: "kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp") pod "network-check-target-rsgg6" (UID: "0a17e4ac-d603-4db3-9d72-578389bf0db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:10.835024 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:10.834940    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:10.835176 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:10.835076    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:10.835176 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:10.835120    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:10.835301 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:10.835220    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:11.834264 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:11.834094    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:11.834264 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:11.834247    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:12.834088 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:12.834050    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:12.834331 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:12.834069    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:12.834331 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:12.834215    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:12.834331 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:12.834302    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:13.833169 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:13.833133    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:13.833326 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:13.833256    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:14.833554 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:14.833515    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:14.833976 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:14.833517    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:14.833976 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:14.833616    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:14.833976 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:14.833742    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:15.833683 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:15.833653    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:15.834082 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:15.833775    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:16.833438 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:16.833407    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:16.833585 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:16.833524    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:16.833585 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:16.833572    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:16.833675 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:16.833659    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:17.833494 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:17.833458    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:17.834013 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:17.833591    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:18.553558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:18.553522    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:18.553744 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:18.553593    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:18.553744 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:18.553716    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:18.553867 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:18.553796    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret podName:11460f81-a3fc-4a03-aa3f-79dc61b95a4a nodeName:}" failed. No retries permitted until 2026-09-09 10:26:34.553778618 +0000 UTC m=+34.217417587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret") pod "global-pull-secret-syncer-crdlr" (UID: "11460f81-a3fc-4a03-aa3f-79dc61b95a4a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:18.553867 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:18.553716    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:18.553867 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:18.553828    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs podName:6969769c-d30e-4cbb-afc5-9df38c741545 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:34.55382178 +0000 UTC m=+34.217460731 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs") pod "network-metrics-daemon-llgq5" (UID: "6969769c-d30e-4cbb-afc5-9df38c741545") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:18.654706 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:18.654664    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9rp\" (UniqueName: \"kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp\") pod \"network-check-target-rsgg6\" (UID: \"0a17e4ac-d603-4db3-9d72-578389bf0db5\") " pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:18.654892 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:18.654868    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 10:26:18.654892 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:18.654893    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 10:26:18.655003 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:18.654905    2570 projected.go:196] Error preparing data for projected volume kube-api-access-tm9rp for pod openshift-network-diagnostics/network-check-target-rsgg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:18.655003 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:18.654974    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp podName:0a17e4ac-d603-4db3-9d72-578389bf0db5 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:34.654952394 +0000 UTC m=+34.318591366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm9rp" (UniqueName: "kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp") pod "network-check-target-rsgg6" (UID: "0a17e4ac-d603-4db3-9d72-578389bf0db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:18.833957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:18.833866    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:18.833957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:18.833923    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:18.834338 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:18.833993    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:18.834338 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:18.834118    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:19.833235 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:19.833199    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:19.833414 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:19.833333    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:20.836240 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:20.836212    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:20.836642 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:20.836327    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:20.836642 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:20.836405    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:20.836642 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:20.836505    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:21.833586 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:21.833557    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:21.833737 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:21.833659    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:22.833947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.833288    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:22.833947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.833303    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:22.833947 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:22.833636    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:22.833947 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:22.833753    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:22.909300 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.907048    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4qlxw" event={"ID":"e24f48e4-042d-469d-a94a-7bb0451cebf7","Type":"ContainerStarted","Data":"87188cad1524cb7171e583d1825b2cd55255880339af035608e038b1dee4cfc5"}
Sep 09 10:26:22.925661 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.925245    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4t6qk" event={"ID":"9fd12634-040b-49e0-b16b-30c8ac321311","Type":"ContainerStarted","Data":"07a2bde84d5df327f33818f6795602863e8dd225ac700a4efd6bf70cde6959d4"}
Sep 09 10:26:22.929563 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.929533    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sh98r" event={"ID":"914c4cb9-1867-478b-8e4e-39185d5fc46d","Type":"ContainerStarted","Data":"1f0a9ef0d61cd1c88002d20c696792f7d0ff84ee86199f3a12fc972558fbf824"}
Sep 09 10:26:22.932405 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.932373    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-193.ec2.internal" event={"ID":"046071a0f74eb379e298ca5ffe8a80f1","Type":"ContainerStarted","Data":"85eb6f02d4c54ccf4094a2141df82596f33bd3c4fa1734e7a6362bee1329cde5"}
Sep 09 10:26:22.934279 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.934236    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4qlxw" podStartSLOduration=2.364361134 podStartE2EDuration="21.934221098s" totalImagesPullingTime="19.569859964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:22.933854775 +0000 UTC m=+22.597493748" watchObservedRunningTime="2026-09-09 10:26:22.934221098 +0000 UTC m=+22.597860071"
Sep 09 10:26:22.938550 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.938529    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:26:22.938990 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.938907    2570 generic.go:356] "Generic (PLEG): container finished" podID="8b797865-8296-4715-b5cb-69bb3f884b62" containerID="1cbf9baef5cdee277ca2e5cc6f75c624c71bd8edfde1816073c0db8e56919794" exitCode=1
Sep 09 10:26:22.939094 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.938997    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn" event={"ID":"8b797865-8296-4715-b5cb-69bb3f884b62","Type":"ContainerStarted","Data":"24a0e839ba403cce30fe2e7ac065865099888f1638503c5467d888218b8d6ead"}
Sep 09 10:26:22.939153 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.939099    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn" event={"ID":"8b797865-8296-4715-b5cb-69bb3f884b62","Type":"ContainerStarted","Data":"408d4f56ff53f0ab940a9998e746b18c56c12791584d6487c0ebc2f019f55a86"}
Sep 09 10:26:22.939204 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.939160    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn" event={"ID":"8b797865-8296-4715-b5cb-69bb3f884b62","Type":"ContainerStarted","Data":"a0b9ec59868a079b8a02125ea5d984b363ac70d056ce0b1ba9aa797c9f6d7c7f"}
Sep 09 10:26:22.939257 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.939243    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn" event={"ID":"8b797865-8296-4715-b5cb-69bb3f884b62","Type":"ContainerDied","Data":"1cbf9baef5cdee277ca2e5cc6f75c624c71bd8edfde1816073c0db8e56919794"}
Sep 09 10:26:22.939655 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.939317    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn" event={"ID":"8b797865-8296-4715-b5cb-69bb3f884b62","Type":"ContainerStarted","Data":"61bcc4a3e1c3e46c17e8e7c03c8c03078a8a1a8f34c7cb062a42f5f97763f72a"}
Sep 09 10:26:22.949461 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:22.949295    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-193.ec2.internal" podStartSLOduration=20.949279636 podStartE2EDuration="20.949279636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:22.948853204 +0000 UTC m=+22.612492178" watchObservedRunningTime="2026-09-09 10:26:22.949279636 +0000 UTC m=+22.612918609"
Sep 09 10:26:23.000946 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.000782    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4t6qk" podStartSLOduration=2.888370831 podStartE2EDuration="22.000750001s" totalImagesPullingTime="19.11237917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:22.965339451 +0000 UTC m=+22.628978425" watchObservedRunningTime="2026-09-09 10:26:23.000750001 +0000 UTC m=+22.664388975"
Sep 09 10:26:23.001083 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.000983    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sh98r" podStartSLOduration=2.857225103 podStartE2EDuration="22.00097669s" totalImagesPullingTime="19.143751587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:23.000726336 +0000 UTC m=+22.664365341" watchObservedRunningTime="2026-09-09 10:26:23.00097669 +0000 UTC m=+22.664615674"
Sep 09 10:26:23.065720 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.065673    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5j47k" podStartSLOduration=2.946435343 podStartE2EDuration="22.065658932s" totalImagesPullingTime="19.119223589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:23.031533845 +0000 UTC m=+22.695172819" watchObservedRunningTime="2026-09-09 10:26:23.065658932 +0000 UTC m=+22.729297906"
Sep 09 10:26:23.833194 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.833174    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:23.833315 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:23.833260    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:23.860085 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.860056    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4t6qk"
Sep 09 10:26:23.860560 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.860538    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4t6qk"
Sep 09 10:26:23.881339 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.881291    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dh892" podStartSLOduration=10.038437842 podStartE2EDuration="21.881276054s" totalImagesPullingTime="11.842838212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:23.092453059 +0000 UTC m=+22.756092033" watchObservedRunningTime="2026-09-09 10:26:23.881276054 +0000 UTC m=+23.544915032"
Sep 09 10:26:23.919710 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.919689    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 10:26:23.945390 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.945363    2570 generic.go:356] "Generic (PLEG): container finished" podID="9a4115943f85345d07176841ce75f166" containerID="59e79209eca3686373aac121e8f24cae667b41f598fa8be734ecce3df9d9604a" exitCode=0
Sep 09 10:26:23.945492 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.945429    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal" event={"ID":"9a4115943f85345d07176841ce75f166","Type":"ContainerDied","Data":"59e79209eca3686373aac121e8f24cae667b41f598fa8be734ecce3df9d9604a"}
Sep 09 10:26:23.946861 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.946839    2570 generic.go:356] "Generic (PLEG): container finished" podID="30ae792d-a70e-4cf5-8d08-cee325dbd625" containerID="d6ab3a232e352f042db5495c95cae13189400a5c0ea2c78c8df104b8aea543f4" exitCode=0
Sep 09 10:26:23.946956 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.946912    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlj8h" event={"ID":"30ae792d-a70e-4cf5-8d08-cee325dbd625","Type":"ContainerDied","Data":"d6ab3a232e352f042db5495c95cae13189400a5c0ea2c78c8df104b8aea543f4"}
Sep 09 10:26:23.948429 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.948399    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5j47k" event={"ID":"55711478-e733-4bdb-a947-0b6f37cc3f75","Type":"ContainerStarted","Data":"ffd8d1b2238bc9684a26581fb101669f05569f0f2a357a1ff974f265779896a9"}
Sep 09 10:26:23.950032 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.949977    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qpgmv" event={"ID":"39947d46-b67f-4338-b764-431cbb707b86","Type":"ContainerStarted","Data":"c21e0df80789fb1785926c27af6c7e592460b32dcb217162dbbdc65aae9aa057"}
Sep 09 10:26:23.951389 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.951371    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dh892" event={"ID":"935d141b-8ba9-42a3-9a46-91f75e24a0bf","Type":"ContainerStarted","Data":"0a618d24f10d1f03136a104b9e29bb8ebf77b7c416c13a468ab1cf375d88f53d"}
Sep 09 10:26:23.954394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.954377    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:26:23.954736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.954720    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn" event={"ID":"8b797865-8296-4715-b5cb-69bb3f884b62","Type":"ContainerStarted","Data":"b0428261af7d78171281e3b4eabedf5cb3f59f583dc6b577841d29a4f25c9dcd"}
Sep 09 10:26:23.956553 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.956532    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn" event={"ID":"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59","Type":"ContainerStarted","Data":"ddfaf5c2f256a47cd4fb7bdd8553c132bfc45b29786336d10da7908e81738141"}
Sep 09 10:26:23.956647 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.956582    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn" event={"ID":"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59","Type":"ContainerStarted","Data":"54211061a02841a559603539c75248fb2e81902f18c87d7c8b2b68b35f6a6116"}
Sep 09 10:26:23.957044 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.957023    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4t6qk"
Sep 09 10:26:23.957444 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:23.957429    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4t6qk"
Sep 09 10:26:24.003895 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:24.003849    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qpgmv" podStartSLOduration=3.876303187 podStartE2EDuration="23.003834279s" totalImagesPullingTime="19.127531092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:24.00299946 +0000 UTC m=+23.666638434" watchObservedRunningTime="2026-09-09 10:26:24.003834279 +0000 UTC m=+23.667473252"
Sep 09 10:26:24.782834 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:24.782684    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T10:26:23.91970825Z","UUID":"5742d823-52f5-4d98-abe6-f615ead61a84","Handler":null,"Name":"","Endpoint":""}
Sep 09 10:26:24.784211 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:24.784189    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 10:26:24.784356 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:24.784224    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 10:26:24.833886 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:24.833857    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:24.834042 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:24.833858    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:24.834042 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:24.833985    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:24.834162 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:24.834044    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:24.961267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:24.961200    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal" event={"ID":"9a4115943f85345d07176841ce75f166","Type":"ContainerStarted","Data":"0dd625e994faf47891d184eb1c50106fd5c8e1c008b8538e16b417190b5f68af"}
Sep 09 10:26:24.979735 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:24.979685    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-193.ec2.internal" podStartSLOduration=22.979667216 podStartE2EDuration="22.979667216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:24.979567284 +0000 UTC m=+24.643206257" watchObservedRunningTime="2026-09-09 10:26:24.979667216 +0000 UTC m=+24.643306191"
Sep 09 10:26:25.833841 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:25.833807    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:25.834001 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:25.833955    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:25.968220 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:25.968191    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:26:25.968781 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:25.968733    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn" event={"ID":"8b797865-8296-4715-b5cb-69bb3f884b62","Type":"ContainerStarted","Data":"36eb343fd0a0143373cbb85783cbe6c91a140041de262fc4d3071c84e094c6e0"}
Sep 09 10:26:26.833529 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:26.833340    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:26.833705 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:26.833340    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:26.833705 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:26.833635    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:26.833850 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:26.833744    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:26.974107 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:26.974066    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn" event={"ID":"ce7dfdb6-b2bd-4d77-94fa-b6ee57306d59","Type":"ContainerStarted","Data":"4871e6c87294c61a1bdd2e72a5e5843fbe291fe83d53d2c86fcee9db6a2dbfd8"}
Sep 09 10:26:26.995805 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:26.995739    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6pwn" podStartSLOduration=15.21727102 podStartE2EDuration="25.995719629s" totalImagesPullingTime="10.778448609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:26.995128984 +0000 UTC m=+26.658767964" watchObservedRunningTime="2026-09-09 10:26:26.995719629 +0000 UTC m=+26.659358604"
Sep 09 10:26:27.833530 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:27.833500    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:27.833692 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:27.833600    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:27.979126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:27.979104    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:26:27.979493 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:27.979400    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn" event={"ID":"8b797865-8296-4715-b5cb-69bb3f884b62","Type":"ContainerStarted","Data":"343c14c3138777e1cf6763164f04731c3b01d9876aae0a07cd05fd5d47e8f15d"}
Sep 09 10:26:27.979971 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:27.979953    2570 scope.go:117] "RemoveContainer" containerID="1cbf9baef5cdee277ca2e5cc6f75c624c71bd8edfde1816073c0db8e56919794"
Sep 09 10:26:28.833643 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:28.833610    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:28.833916 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:28.833725    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:28.833916 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:28.833781    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:28.833916 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:28.833884    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:28.984922 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:28.984896    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:26:28.985320 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:28.985281    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn" event={"ID":"8b797865-8296-4715-b5cb-69bb3f884b62","Type":"ContainerStarted","Data":"49a642426b3ab43644d40f011dca6fe57a665002e755916180dcaedddd1e0e3d"}
Sep 09 10:26:28.985381 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:28.985360    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:28.985569 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:28.985528    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:28.985678 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:28.985589    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:28.987411 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:28.987387    2570 generic.go:356] "Generic (PLEG): container finished" podID="30ae792d-a70e-4cf5-8d08-cee325dbd625" containerID="42039e4daadcf9f45db5950fecc6bfd863fade115069e7f5595b700244e73d06" exitCode=0
Sep 09 10:26:28.987503 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:28.987423    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlj8h" event={"ID":"30ae792d-a70e-4cf5-8d08-cee325dbd625","Type":"ContainerDied","Data":"42039e4daadcf9f45db5950fecc6bfd863fade115069e7f5595b700244e73d06"}
Sep 09 10:26:29.000651 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:29.000631    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:29.000918 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:29.000903    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:26:29.017747 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:29.017710    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn" podStartSLOduration=8.906015765 podStartE2EDuration="28.017697942s" totalImagesPullingTime="19.111682177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:29.016594961 +0000 UTC m=+28.680233935" watchObservedRunningTime="2026-09-09 10:26:29.017697942 +0000 UTC m=+28.681336953"
Sep 09 10:26:29.833831 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:29.833797    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:29.833939 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:29.833902    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:29.938291 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:29.938216    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-crdlr"]
Sep 09 10:26:29.940234 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:29.940202    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rsgg6"]
Sep 09 10:26:29.940542 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:29.940356    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:29.940542 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:29.940461    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:29.941176 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:29.941153    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-llgq5"]
Sep 09 10:26:29.941297 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:29.941244    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:29.941351 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:29.941323    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:29.992023 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:29.991990    2570 generic.go:356] "Generic (PLEG): container finished" podID="30ae792d-a70e-4cf5-8d08-cee325dbd625" containerID="6adfe7032c110cd09fb3c78184e5705c6085e547bfb60242445fb49fddefd33d" exitCode=0
Sep 09 10:26:29.992660 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:29.992061    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlj8h" event={"ID":"30ae792d-a70e-4cf5-8d08-cee325dbd625","Type":"ContainerDied","Data":"6adfe7032c110cd09fb3c78184e5705c6085e547bfb60242445fb49fddefd33d"}
Sep 09 10:26:29.992660 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:29.992081    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:29.992660 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:29.992165    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:30.996663 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:30.996634    2570 generic.go:356] "Generic (PLEG): container finished" podID="30ae792d-a70e-4cf5-8d08-cee325dbd625" containerID="e6a5fd1d7f7e37f131eaff5e54866be52aa0eaed4ddf2dd4cdd3216e7a3e45c4" exitCode=0
Sep 09 10:26:30.997014 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:30.996711    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlj8h" event={"ID":"30ae792d-a70e-4cf5-8d08-cee325dbd625","Type":"ContainerDied","Data":"e6a5fd1d7f7e37f131eaff5e54866be52aa0eaed4ddf2dd4cdd3216e7a3e45c4"}
Sep 09 10:26:31.833400 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:31.833217    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:31.833563 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:31.833247    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:31.833563 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:31.833492    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:31.833563 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:31.833255    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:31.833726 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:31.833562    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:31.833726 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:31.833638    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:33.833708 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:33.833672    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:33.833708 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:33.833701    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:33.834153 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:33.833672    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:33.834153 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:33.833803    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsgg6" podUID="0a17e4ac-d603-4db3-9d72-578389bf0db5"
Sep 09 10:26:33.834153 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:33.833874    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crdlr" podUID="11460f81-a3fc-4a03-aa3f-79dc61b95a4a"
Sep 09 10:26:33.834153 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:33.833958    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:26:34.575497 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:34.575458    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:34.575695 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:34.575532    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:34.575695 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:34.575614    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:34.576029 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:34.576006    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret podName:11460f81-a3fc-4a03-aa3f-79dc61b95a4a nodeName:}" failed. No retries permitted until 2026-09-09 10:27:06.575670959 +0000 UTC m=+66.239309910 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret") pod "global-pull-secret-syncer-crdlr" (UID: "11460f81-a3fc-4a03-aa3f-79dc61b95a4a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:34.576029 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:34.575854    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:34.576236 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:34.576087    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs podName:6969769c-d30e-4cbb-afc5-9df38c741545 nodeName:}" failed. No retries permitted until 2026-09-09 10:27:06.576060005 +0000 UTC m=+66.239698978 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs") pod "network-metrics-daemon-llgq5" (UID: "6969769c-d30e-4cbb-afc5-9df38c741545") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:34.676684 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:34.676651    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9rp\" (UniqueName: \"kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp\") pod \"network-check-target-rsgg6\" (UID: \"0a17e4ac-d603-4db3-9d72-578389bf0db5\") " pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:34.676877 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:34.676857    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 10:26:34.676944 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:34.676885    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 10:26:34.676944 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:34.676898    2570 projected.go:196] Error preparing data for projected volume kube-api-access-tm9rp for pod openshift-network-diagnostics/network-check-target-rsgg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:34.677038 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:34.676966    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp podName:0a17e4ac-d603-4db3-9d72-578389bf0db5 nodeName:}" failed. No retries permitted until 2026-09-09 10:27:06.676944327 +0000 UTC m=+66.340583299 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm9rp" (UniqueName: "kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp") pod "network-check-target-rsgg6" (UID: "0a17e4ac-d603-4db3-9d72-578389bf0db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:35.138377 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.138342    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-193.ec2.internal" event="NodeReady"
Sep 09 10:26:35.138806 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.138475    2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 10:26:35.213081 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.213047    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5llp7"]
Sep 09 10:26:35.227432 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.227402    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-75ncn"]
Sep 09 10:26:35.227596 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.227540    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5llp7"
Sep 09 10:26:35.231674 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.231495    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wx2nx\""
Sep 09 10:26:35.231674 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.231570    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 10:26:35.231674 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.231664    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 10:26:35.232027 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.232010    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 10:26:35.235796 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.235776    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5llp7"]
Sep 09 10:26:35.235895 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.235846    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:35.238557 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.238535    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 10:26:35.238669 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.238648    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-272r8\""
Sep 09 10:26:35.238901 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.238887    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 10:26:35.245990 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.245850    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-75ncn"]
Sep 09 10:26:35.383270 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.383199    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-tmp-dir\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:35.383270 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.383233    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-config-volume\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:35.383270 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.383251    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:35.383485 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.383368    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d95d07c-cf59-43f5-a825-a0744b99eafb-cert\") pod \"ingress-canary-5llp7\" (UID: \"6d95d07c-cf59-43f5-a825-a0744b99eafb\") " pod="openshift-ingress-canary/ingress-canary-5llp7"
Sep 09 10:26:35.383485 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.383400    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnchb\" (UniqueName: \"kubernetes.io/projected/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-kube-api-access-tnchb\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:35.383485 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.383426    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27bk7\" (UniqueName: \"kubernetes.io/projected/6d95d07c-cf59-43f5-a825-a0744b99eafb-kube-api-access-27bk7\") pod \"ingress-canary-5llp7\" (UID: \"6d95d07c-cf59-43f5-a825-a0744b99eafb\") " pod="openshift-ingress-canary/ingress-canary-5llp7"
Sep 09 10:26:35.484072 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.484039    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-tmp-dir\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:35.484072 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.484076    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-config-volume\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:35.484279 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.484092    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:35.484279 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.484146    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d95d07c-cf59-43f5-a825-a0744b99eafb-cert\") pod \"ingress-canary-5llp7\" (UID: \"6d95d07c-cf59-43f5-a825-a0744b99eafb\") " pod="openshift-ingress-canary/ingress-canary-5llp7"
Sep 09 10:26:35.484279 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.484165    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnchb\" (UniqueName: \"kubernetes.io/projected/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-kube-api-access-tnchb\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:35.484279 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.484184    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27bk7\" (UniqueName: \"kubernetes.io/projected/6d95d07c-cf59-43f5-a825-a0744b99eafb-kube-api-access-27bk7\") pod \"ingress-canary-5llp7\" (UID: \"6d95d07c-cf59-43f5-a825-a0744b99eafb\") " pod="openshift-ingress-canary/ingress-canary-5llp7"
Sep 09 10:26:35.484417 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:35.484319    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:26:35.484417 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:35.484394    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls podName:1c1c0fa9-ce40-473e-a699-540afd0a9ed9 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:35.984373274 +0000 UTC m=+35.648012245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls") pod "dns-default-75ncn" (UID: "1c1c0fa9-ce40-473e-a699-540afd0a9ed9") : secret "dns-default-metrics-tls" not found
Sep 09 10:26:35.484417 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.484406    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-tmp-dir\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:35.484603 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.484585    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-config-volume\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:35.488226 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.488204    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d95d07c-cf59-43f5-a825-a0744b99eafb-cert\") pod \"ingress-canary-5llp7\" (UID: \"6d95d07c-cf59-43f5-a825-a0744b99eafb\") " pod="openshift-ingress-canary/ingress-canary-5llp7"
Sep 09 10:26:35.493171 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.493148    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnchb\" (UniqueName: \"kubernetes.io/projected/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-kube-api-access-tnchb\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:35.493343 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.493327    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27bk7\" (UniqueName: \"kubernetes.io/projected/6d95d07c-cf59-43f5-a825-a0744b99eafb-kube-api-access-27bk7\") pod \"ingress-canary-5llp7\" (UID: \"6d95d07c-cf59-43f5-a825-a0744b99eafb\") " pod="openshift-ingress-canary/ingress-canary-5llp7"
Sep 09 10:26:35.539249 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.539219    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5llp7"
Sep 09 10:26:35.833802 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.833635    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:26:35.833917 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.833644    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:26:35.833969 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.833644    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:26:35.836690 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.836669    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 10:26:35.836690 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.836678    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 10:26:35.836858 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.836693    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hq5pc\""
Sep 09 10:26:35.836858 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.836686    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ffrc8\""
Sep 09 10:26:35.837065 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.837043    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 10:26:35.837241 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.837225    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 10:26:35.866061 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.866027    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5llp7"]
Sep 09 10:26:35.870491 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:26:35.870463    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d95d07c_cf59_43f5_a825_a0744b99eafb.slice/crio-e60c4a66678137b83d6a67fe00dd3bbc680f8613c0237e73b9b81f9667030ed3 WatchSource:0}: Error finding container e60c4a66678137b83d6a67fe00dd3bbc680f8613c0237e73b9b81f9667030ed3: Status 404 returned error can't find the container with id e60c4a66678137b83d6a67fe00dd3bbc680f8613c0237e73b9b81f9667030ed3
Sep 09 10:26:35.987292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:35.987248    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:35.987448 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:35.987371    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:26:35.987448 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:35.987437    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls podName:1c1c0fa9-ce40-473e-a699-540afd0a9ed9 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:36.987419494 +0000 UTC m=+36.651058446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls") pod "dns-default-75ncn" (UID: "1c1c0fa9-ce40-473e-a699-540afd0a9ed9") : secret "dns-default-metrics-tls" not found
Sep 09 10:26:36.011916 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:36.011889    2570 generic.go:356] "Generic (PLEG): container finished" podID="30ae792d-a70e-4cf5-8d08-cee325dbd625" containerID="18b5b29aa1f0d559a17c2a2b12ad8e65e17f8cca604e447594f903d6d591b324" exitCode=0
Sep 09 10:26:36.012053 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:36.011962    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlj8h" event={"ID":"30ae792d-a70e-4cf5-8d08-cee325dbd625","Type":"ContainerDied","Data":"18b5b29aa1f0d559a17c2a2b12ad8e65e17f8cca604e447594f903d6d591b324"}
Sep 09 10:26:36.013151 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:36.013119    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5llp7" event={"ID":"6d95d07c-cf59-43f5-a825-a0744b99eafb","Type":"ContainerStarted","Data":"e60c4a66678137b83d6a67fe00dd3bbc680f8613c0237e73b9b81f9667030ed3"}
Sep 09 10:26:36.471448 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:36.471371    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:36.563972 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:36.563921    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:36.796469 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:36.796435    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:36.896089 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:36.896057    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:36.995630 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:36.995600    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:36.995761 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:36.995715    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:26:36.995846 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:36.995783    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls podName:1c1c0fa9-ce40-473e-a699-540afd0a9ed9 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:38.995752815 +0000 UTC m=+38.659391766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls") pod "dns-default-75ncn" (UID: "1c1c0fa9-ce40-473e-a699-540afd0a9ed9") : secret "dns-default-metrics-tls" not found
Sep 09 10:26:37.018063 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:37.018034    2570 generic.go:356] "Generic (PLEG): container finished" podID="30ae792d-a70e-4cf5-8d08-cee325dbd625" containerID="24a03a0d98397399ff45d10823203aeb37074e3879d950187b65cd08e57efd34" exitCode=0
Sep 09 10:26:37.018187 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:37.018067    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlj8h" event={"ID":"30ae792d-a70e-4cf5-8d08-cee325dbd625","Type":"ContainerDied","Data":"24a03a0d98397399ff45d10823203aeb37074e3879d950187b65cd08e57efd34"}
Sep 09 10:26:38.024654 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:38.024616    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlj8h" event={"ID":"30ae792d-a70e-4cf5-8d08-cee325dbd625","Type":"ContainerStarted","Data":"ff70a41e457124f3d07fce49904ba608f6d7aea78cebc19b3e383cfa8fe215bf"}
Sep 09 10:26:38.190045 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:38.190004    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:38.285681 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:38.285537    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:39.013393 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:39.013363    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:39.013569 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:39.013518    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:26:39.013637 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:39.013592    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls podName:1c1c0fa9-ce40-473e-a699-540afd0a9ed9 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:43.013570319 +0000 UTC m=+42.677209272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls") pod "dns-default-75ncn" (UID: "1c1c0fa9-ce40-473e-a699-540afd0a9ed9") : secret "dns-default-metrics-tls" not found
Sep 09 10:26:39.328033 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:39.327949    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:39.430341 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:39.430304    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:40.030667 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:40.030636    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5llp7" event={"ID":"6d95d07c-cf59-43f5-a825-a0744b99eafb","Type":"ContainerStarted","Data":"3202a9544f197f7149bdb469e8d55353370594b855bd2374d02e3f68bbfbc814"}
Sep 09 10:26:40.050739 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:40.050670    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vlj8h" podStartSLOduration=9.108623939 podStartE2EDuration="38.050654342s" totalImagesPullingTime="28.942030403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:38.058540571 +0000 UTC m=+37.722179546" watchObservedRunningTime="2026-09-09 10:26:40.050654342 +0000 UTC m=+39.714293316"
Sep 09 10:26:40.050952 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:40.050933    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5llp7" podStartSLOduration=1.817255058 podStartE2EDuration="5.050924983s" totalImagesPullingTime="3.233669925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:40.049460432 +0000 UTC m=+39.713099407" watchObservedRunningTime="2026-09-09 10:26:40.050924983 +0000 UTC m=+39.714563956"
Sep 09 10:26:40.679852 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:40.679821    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:42.701529 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:42.701499    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:42.769803 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:42.769732    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:42.887521 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:42.887485    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:42.989322 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:42.989291    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:43.044860 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:43.044831    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:43.045028 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:43.045004    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:26:43.045094 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:43.045077    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls podName:1c1c0fa9-ce40-473e-a699-540afd0a9ed9 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:51.045054993 +0000 UTC m=+50.708693962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls") pod "dns-default-75ncn" (UID: "1c1c0fa9-ce40-473e-a699-540afd0a9ed9") : secret "dns-default-metrics-tls" not found
Sep 09 10:26:51.091568 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:51.091539    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:26:51.092008 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:51.091687    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:26:51.092008 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:26:51.091757    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls podName:1c1c0fa9-ce40-473e-a699-540afd0a9ed9 nodeName:}" failed. No retries permitted until 2026-09-09 10:27:07.091742082 +0000 UTC m=+66.755381039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls") pod "dns-default-75ncn" (UID: "1c1c0fa9-ce40-473e-a699-540afd0a9ed9") : secret "dns-default-metrics-tls" not found
Sep 09 10:26:57.773615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:57.773581    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:57.835883 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:57.835853    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:26:59.141527 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:26:59.141490    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:27:00.708802 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:00.708775    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:27:00.824863 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:00.824835    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:27:01.006951 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:01.006925    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs8qn"
Sep 09 10:27:01.153861 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:01.153827    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:27:01.273572 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:01.273496    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:27:06.589369 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:06.589330    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:27:06.589829 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:06.589386    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:27:06.592528 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:06.592510    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 10:27:06.592903 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:06.592892    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 10:27:06.599848 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:27:06.599827    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 10:27:06.599930 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:27:06.599893    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs podName:6969769c-d30e-4cbb-afc5-9df38c741545 nodeName:}" failed. No retries permitted until 2026-09-09 10:28:10.599877946 +0000 UTC m=+130.263516899 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs") pod "network-metrics-daemon-llgq5" (UID: "6969769c-d30e-4cbb-afc5-9df38c741545") : secret "metrics-daemon-secret" not found
Sep 09 10:27:06.602788 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:06.602745    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/11460f81-a3fc-4a03-aa3f-79dc61b95a4a-original-pull-secret\") pod \"global-pull-secret-syncer-crdlr\" (UID: \"11460f81-a3fc-4a03-aa3f-79dc61b95a4a\") " pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:27:06.690518 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:06.690492    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9rp\" (UniqueName: \"kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp\") pod \"network-check-target-rsgg6\" (UID: \"0a17e4ac-d603-4db3-9d72-578389bf0db5\") " pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:27:06.693725 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:06.693708    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 10:27:06.704058 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:06.704040    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 10:27:06.713686 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:06.713657    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm9rp\" (UniqueName: \"kubernetes.io/projected/0a17e4ac-d603-4db3-9d72-578389bf0db5-kube-api-access-tm9rp\") pod \"network-check-target-rsgg6\" (UID: \"0a17e4ac-d603-4db3-9d72-578389bf0db5\") " pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:27:06.781876 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:06.781838    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crdlr"
Sep 09 10:27:06.789068 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:06.789045    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ffrc8\""
Sep 09 10:27:06.796751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:06.796730    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:27:07.084721 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:07.084695    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-crdlr"]
Sep 09 10:27:07.087238 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:27:07.087212    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11460f81_a3fc_4a03_aa3f_79dc61b95a4a.slice/crio-145c1387d6dd7f083d5455bf7a6434b2ccccea1bc7601cca0a5c393783090822 WatchSource:0}: Error finding container 145c1387d6dd7f083d5455bf7a6434b2ccccea1bc7601cca0a5c393783090822: Status 404 returned error can't find the container with id 145c1387d6dd7f083d5455bf7a6434b2ccccea1bc7601cca0a5c393783090822
Sep 09 10:27:07.093294 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:07.093232    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-crdlr" event={"ID":"11460f81-a3fc-4a03-aa3f-79dc61b95a4a","Type":"ContainerStarted","Data":"145c1387d6dd7f083d5455bf7a6434b2ccccea1bc7601cca0a5c393783090822"}
Sep 09 10:27:07.094541 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:07.094522    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:27:07.094655 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:27:07.094643    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:27:07.094711 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:27:07.094702    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls podName:1c1c0fa9-ce40-473e-a699-540afd0a9ed9 nodeName:}" failed. No retries permitted until 2026-09-09 10:27:39.094684755 +0000 UTC m=+98.758323706 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls") pod "dns-default-75ncn" (UID: "1c1c0fa9-ce40-473e-a699-540afd0a9ed9") : secret "dns-default-metrics-tls" not found
Sep 09 10:27:07.109712 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:07.109688    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rsgg6"]
Sep 09 10:27:07.113186 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:27:07.113148    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a17e4ac_d603_4db3_9d72_578389bf0db5.slice/crio-5932fd3481511a1eade2beedfe0a0cdc644211bf56e503df4322140479e4ffea WatchSource:0}: Error finding container 5932fd3481511a1eade2beedfe0a0cdc644211bf56e503df4322140479e4ffea: Status 404 returned error can't find the container with id 5932fd3481511a1eade2beedfe0a0cdc644211bf56e503df4322140479e4ffea
Sep 09 10:27:08.097525 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:08.097487    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rsgg6" event={"ID":"0a17e4ac-d603-4db3-9d72-578389bf0db5","Type":"ContainerStarted","Data":"5932fd3481511a1eade2beedfe0a0cdc644211bf56e503df4322140479e4ffea"}
Sep 09 10:27:11.600558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:11.600530    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:27:11.721426 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:11.721333    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:27:12.109199 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:12.109158    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-crdlr" event={"ID":"11460f81-a3fc-4a03-aa3f-79dc61b95a4a","Type":"ContainerStarted","Data":"673a81b705e9d0aa2982f94e4f239cfae7cc8afa4dec332117b05e7853375961"}
Sep 09 10:27:12.110759 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:12.110733    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rsgg6" event={"ID":"0a17e4ac-d603-4db3-9d72-578389bf0db5","Type":"ContainerStarted","Data":"c7f302b69f1b8cab388866f524a1b6967f528afe674a51d6abf6b66e3c67f48b"}
Sep 09 10:27:12.110900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:12.110885    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:27:12.130579 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:12.130532    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-crdlr" podStartSLOduration=65.827198405 podStartE2EDuration="1m10.130517367s" totalImagesPullingTime="4.303318962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:27:12.129159683 +0000 UTC m=+71.792798656" watchObservedRunningTime="2026-09-09 10:27:12.130517367 +0000 UTC m=+71.794156392"
Sep 09 10:27:12.156970 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:12.156928    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rsgg6" podStartSLOduration=66.882295611 podStartE2EDuration="1m11.156912343s" totalImagesPullingTime="4.274616732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:27:12.156330951 +0000 UTC m=+71.819969924" watchObservedRunningTime="2026-09-09 10:27:12.156912343 +0000 UTC m=+71.820551317"
Sep 09 10:27:39.112440 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:39.112316    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:27:39.112440 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:27:39.112432    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:27:39.112851 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:27:39.112495    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls podName:1c1c0fa9-ce40-473e-a699-540afd0a9ed9 nodeName:}" failed. No retries permitted until 2026-09-09 10:28:43.112481647 +0000 UTC m=+162.776120598 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls") pod "dns-default-75ncn" (UID: "1c1c0fa9-ce40-473e-a699-540afd0a9ed9") : secret "dns-default-metrics-tls" not found
Sep 09 10:27:43.115491 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:27:43.115457    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rsgg6"
Sep 09 10:28:01.205905 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.205871    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"]
Sep 09 10:28:01.207737 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.207722    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:28:01.210703 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.210678    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 10:28:01.210819 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.210704    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 10:28:01.211876 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.211728    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 10:28:01.211876 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.211749    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 10:28:01.221419 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.221398    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"]
Sep 09 10:28:01.351850 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.351817    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a447e513-d9f3-4354-bcf8-76282e8cd872-klusterlet-config\") pod \"klusterlet-addon-workmgr-565d575bbb-b7ds8\" (UID: \"a447e513-d9f3-4354-bcf8-76282e8cd872\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:28:01.352009 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.351886    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps5fh\" (UniqueName: \"kubernetes.io/projected/a447e513-d9f3-4354-bcf8-76282e8cd872-kube-api-access-ps5fh\") pod \"klusterlet-addon-workmgr-565d575bbb-b7ds8\" (UID: \"a447e513-d9f3-4354-bcf8-76282e8cd872\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:28:01.352009 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.351915    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a447e513-d9f3-4354-bcf8-76282e8cd872-tmp\") pod \"klusterlet-addon-workmgr-565d575bbb-b7ds8\" (UID: \"a447e513-d9f3-4354-bcf8-76282e8cd872\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:28:01.452776 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.452717    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps5fh\" (UniqueName: \"kubernetes.io/projected/a447e513-d9f3-4354-bcf8-76282e8cd872-kube-api-access-ps5fh\") pod \"klusterlet-addon-workmgr-565d575bbb-b7ds8\" (UID: \"a447e513-d9f3-4354-bcf8-76282e8cd872\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:28:01.452856 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.452801    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a447e513-d9f3-4354-bcf8-76282e8cd872-tmp\") pod \"klusterlet-addon-workmgr-565d575bbb-b7ds8\" (UID: \"a447e513-d9f3-4354-bcf8-76282e8cd872\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:28:01.452856 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.452823    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a447e513-d9f3-4354-bcf8-76282e8cd872-klusterlet-config\") pod \"klusterlet-addon-workmgr-565d575bbb-b7ds8\" (UID: \"a447e513-d9f3-4354-bcf8-76282e8cd872\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:28:01.453167 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.453150    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a447e513-d9f3-4354-bcf8-76282e8cd872-tmp\") pod \"klusterlet-addon-workmgr-565d575bbb-b7ds8\" (UID: \"a447e513-d9f3-4354-bcf8-76282e8cd872\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:28:01.455233 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.455205    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a447e513-d9f3-4354-bcf8-76282e8cd872-klusterlet-config\") pod \"klusterlet-addon-workmgr-565d575bbb-b7ds8\" (UID: \"a447e513-d9f3-4354-bcf8-76282e8cd872\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:28:01.461342 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.461291    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps5fh\" (UniqueName: \"kubernetes.io/projected/a447e513-d9f3-4354-bcf8-76282e8cd872-kube-api-access-ps5fh\") pod \"klusterlet-addon-workmgr-565d575bbb-b7ds8\" (UID: \"a447e513-d9f3-4354-bcf8-76282e8cd872\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:28:01.520303 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.520274    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:28:01.787191 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:01.787158    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"]
Sep 09 10:28:01.790950 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:28:01.790923    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda447e513_d9f3_4354_bcf8_76282e8cd872.slice/crio-7fb5abdad83925a6c8ed757695b72667216d6d0b9e3586eba6b5c2090976df9a WatchSource:0}: Error finding container 7fb5abdad83925a6c8ed757695b72667216d6d0b9e3586eba6b5c2090976df9a: Status 404 returned error can't find the container with id 7fb5abdad83925a6c8ed757695b72667216d6d0b9e3586eba6b5c2090976df9a
Sep 09 10:28:02.229436 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:02.229352    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8" event={"ID":"a447e513-d9f3-4354-bcf8-76282e8cd872","Type":"ContainerStarted","Data":"7fb5abdad83925a6c8ed757695b72667216d6d0b9e3586eba6b5c2090976df9a"}
Sep 09 10:28:06.240083 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:06.240045    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8" event={"ID":"a447e513-d9f3-4354-bcf8-76282e8cd872","Type":"ContainerStarted","Data":"e32ff50431916dc4928446fdfa16b3e1ae556f9061c2ac38f41de1980d0d76d1"}
Sep 09 10:28:06.240445 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:06.240234    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:28:06.241807 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:06.241785    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:28:06.257664 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:06.257617    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8" podStartSLOduration=1.396500111 podStartE2EDuration="5.257603074s" totalImagesPullingTime="3.861102963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:28:06.256335933 +0000 UTC m=+125.919974910" watchObservedRunningTime="2026-09-09 10:28:06.257603074 +0000 UTC m=+125.921242047"
Sep 09 10:28:10.619334 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:10.619295    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:28:10.619714 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:28:10.619417    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 10:28:10.619714 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:28:10.619470    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs podName:6969769c-d30e-4cbb-afc5-9df38c741545 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:12.619455754 +0000 UTC m=+252.283094706 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs") pod "network-metrics-daemon-llgq5" (UID: "6969769c-d30e-4cbb-afc5-9df38c741545") : secret "metrics-daemon-secret" not found
Sep 09 10:28:38.246161 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:28:38.246117    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-75ncn" podUID="1c1c0fa9-ce40-473e-a699-540afd0a9ed9"
Sep 09 10:28:38.319264 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:38.319241    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-75ncn"
Sep 09 10:28:38.875525 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:28:38.875486    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:28:43.136279 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:43.136245    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:28:43.136678 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:28:43.136377    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:28:43.136678 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:28:43.136443    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls podName:1c1c0fa9-ce40-473e-a699-540afd0a9ed9 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:45.136425595 +0000 UTC m=+284.800064556 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls") pod "dns-default-75ncn" (UID: "1c1c0fa9-ce40-473e-a699-540afd0a9ed9") : secret "dns-default-metrics-tls" not found
Sep 09 10:28:49.834051 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:28:49.834009    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:29:06.156150 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:29:06.156099    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:06.240783 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.240721    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8" podUID="a447e513-d9f3-4354-bcf8-76282e8cd872" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.8:8000/readyz\": dial tcp 10.134.0.8:8000: connect: connection refused"
Sep 09 10:30:06.529041 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.528958    2570 generic.go:356] "Generic (PLEG): container finished" podID="a447e513-d9f3-4354-bcf8-76282e8cd872" containerID="e32ff50431916dc4928446fdfa16b3e1ae556f9061c2ac38f41de1980d0d76d1" exitCode=1
Sep 09 10:30:06.529041 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.529016    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8" event={"ID":"a447e513-d9f3-4354-bcf8-76282e8cd872","Type":"ContainerDied","Data":"e32ff50431916dc4928446fdfa16b3e1ae556f9061c2ac38f41de1980d0d76d1"}
Sep 09 10:30:06.529360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.529344    2570 scope.go:117] "RemoveContainer" containerID="e32ff50431916dc4928446fdfa16b3e1ae556f9061c2ac38f41de1980d0d76d1"
Sep 09 10:30:06.973095 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.973003    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-75db9dfd44-b2ptj"]
Sep 09 10:30:06.974907 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.974889    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fb4d46669-6v69v"]
Sep 09 10:30:06.975027 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.975011    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:06.976401 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.976385    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"]
Sep 09 10:30:06.976531 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.976514    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:06.978050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.978035    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"
Sep 09 10:30:06.982628 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.982611    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 10:30:06.982723 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.982704    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 10:30:06.982904 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.982886    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 10:30:06.983027 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.982954    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 10:30:06.983027 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.982954    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 10:30:06.983027 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.982985    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 10:30:06.984144 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.984128    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 10:30:06.984296 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.984226    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 10:30:06.984296 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.984249    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-w7rpp\""
Sep 09 10:30:06.984410 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.984389    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 10:30:06.984410 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.984405    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 10:30:06.984513 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.984438    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 10:30:06.984513 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.984505    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gjgqp\""
Sep 09 10:30:06.985118 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.985099    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 10:30:06.985207 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.985136    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 10:30:06.985207 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.985148    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4xzkl\""
Sep 09 10:30:06.989713 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.989694    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 10:30:06.998193 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:06.998175    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"]
Sep 09 10:30:07.001807 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.001789    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb4d46669-6v69v"]
Sep 09 10:30:07.002730 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.002712    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75db9dfd44-b2ptj"]
Sep 09 10:30:07.061669 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.061638    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-67lpw"]
Sep 09 10:30:07.063612 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.063596    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.066783 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.066751    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 10:30:07.066879 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.066787    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 10:30:07.066986 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.066963    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 10:30:07.067100 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.067072    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 10:30:07.067151 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.067075    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-c9btj\""
Sep 09 10:30:07.078758 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.078702    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 10:30:07.083510 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.083487    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-67lpw"]
Sep 09 10:30:07.112158 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112132    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.112251 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112162    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2q2d\" (UniqueName: \"kubernetes.io/projected/5a7fc401-27df-4623-b7e4-9f06886fe86a-kube-api-access-z2q2d\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ggpsk\" (UID: \"5a7fc401-27df-4623-b7e4-9f06886fe86a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"
Sep 09 10:30:07.112251 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112191    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-bound-sa-token\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.112251 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112212    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a7fc401-27df-4623-b7e4-9f06886fe86a-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ggpsk\" (UID: \"5a7fc401-27df-4623-b7e4-9f06886fe86a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"
Sep 09 10:30:07.112251 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112239    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-certificates\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.112390 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112311    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-default-certificate\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.112390 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112351    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe4c67ed-5d54-4d1c-84b4-febd3326c935-image-registry-private-configuration\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.112390 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112371    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-stats-auth\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.112480 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112392    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.112480 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112421    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe4c67ed-5d54-4d1c-84b4-febd3326c935-ca-trust-extracted\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.112480 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112444    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe4c67ed-5d54-4d1c-84b4-febd3326c935-trusted-ca\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.112480 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112457    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.112480 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112471    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lcv2\" (UniqueName: \"kubernetes.io/projected/c824a8cb-67bf-4ee7-bc76-919452663ea1-kube-api-access-7lcv2\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.112701 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112493    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjgv8\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-kube-api-access-bjgv8\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.112701 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112576    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe4c67ed-5d54-4d1c-84b4-febd3326c935-installation-pull-secrets\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.112701 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.112609    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a7fc401-27df-4623-b7e4-9f06886fe86a-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ggpsk\" (UID: \"5a7fc401-27df-4623-b7e4-9f06886fe86a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"
Sep 09 10:30:07.122557 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.122535    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:07.205308 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.205272    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:07.213381 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.213351    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-default-certificate\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.213498 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.213392    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe4c67ed-5d54-4d1c-84b4-febd3326c935-image-registry-private-configuration\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.213498 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.213416    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-stats-auth\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.213602 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.213522    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.213709 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.213550    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55j2l\" (UniqueName: \"kubernetes.io/projected/e16b6c9f-7090-4e95-b0eb-0b07907d992f-kube-api-access-55j2l\") pod \"console-operator-85b454874b-67lpw\" (UID: \"e16b6c9f-7090-4e95-b0eb-0b07907d992f\") " pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.213875 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.213855    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjgv8\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-kube-api-access-bjgv8\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.213954 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.213904    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe4c67ed-5d54-4d1c-84b4-febd3326c935-ca-trust-extracted\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.214023 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:07.214002    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:30:07.214077 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:07.214029    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb4d46669-6v69v: secret "image-registry-tls" not found
Sep 09 10:30:07.214139 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:07.214104    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls podName:fe4c67ed-5d54-4d1c-84b4-febd3326c935 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:07.714075143 +0000 UTC m=+247.377714118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls") pod "image-registry-fb4d46669-6v69v" (UID: "fe4c67ed-5d54-4d1c-84b4-febd3326c935") : secret "image-registry-tls" not found
Sep 09 10:30:07.214200 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.214185    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.214254 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.214230    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.214306 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.214273    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe4c67ed-5d54-4d1c-84b4-febd3326c935-installation-pull-secrets\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.214357 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.214304    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2q2d\" (UniqueName: \"kubernetes.io/projected/5a7fc401-27df-4623-b7e4-9f06886fe86a-kube-api-access-z2q2d\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ggpsk\" (UID: \"5a7fc401-27df-4623-b7e4-9f06886fe86a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"
Sep 09 10:30:07.214357 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.214350    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e16b6c9f-7090-4e95-b0eb-0b07907d992f-config\") pod \"console-operator-85b454874b-67lpw\" (UID: \"e16b6c9f-7090-4e95-b0eb-0b07907d992f\") " pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.214480 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.214457    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe4c67ed-5d54-4d1c-84b4-febd3326c935-ca-trust-extracted\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.215048 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:07.214590    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 10:30:07.215048 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:07.214911    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs podName:c824a8cb-67bf-4ee7-bc76-919452663ea1 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:07.714883165 +0000 UTC m=+247.378522135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs") pod "router-default-75db9dfd44-b2ptj" (UID: "c824a8cb-67bf-4ee7-bc76-919452663ea1") : secret "router-metrics-certs-default" not found
Sep 09 10:30:07.215048 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:07.215015    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle podName:c824a8cb-67bf-4ee7-bc76-919452663ea1 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:07.715002699 +0000 UTC m=+247.378641668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle") pod "router-default-75db9dfd44-b2ptj" (UID: "c824a8cb-67bf-4ee7-bc76-919452663ea1") : configmap references non-existent config key: service-ca.crt
Sep 09 10:30:07.215305 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.214392    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-certificates\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.215305 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.215189    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e16b6c9f-7090-4e95-b0eb-0b07907d992f-serving-cert\") pod \"console-operator-85b454874b-67lpw\" (UID: \"e16b6c9f-7090-4e95-b0eb-0b07907d992f\") " pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.215305 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.215233    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe4c67ed-5d54-4d1c-84b4-febd3326c935-trusted-ca\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.215305 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.215269    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lcv2\" (UniqueName: \"kubernetes.io/projected/c824a8cb-67bf-4ee7-bc76-919452663ea1-kube-api-access-7lcv2\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.215499 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.215416    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e16b6c9f-7090-4e95-b0eb-0b07907d992f-trusted-ca\") pod \"console-operator-85b454874b-67lpw\" (UID: \"e16b6c9f-7090-4e95-b0eb-0b07907d992f\") " pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.215499 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.215461    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a7fc401-27df-4623-b7e4-9f06886fe86a-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ggpsk\" (UID: \"5a7fc401-27df-4623-b7e4-9f06886fe86a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"
Sep 09 10:30:07.216189 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.215529    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-bound-sa-token\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.216189 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.215727    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a7fc401-27df-4623-b7e4-9f06886fe86a-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ggpsk\" (UID: \"5a7fc401-27df-4623-b7e4-9f06886fe86a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"
Sep 09 10:30:07.216189 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.215420    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-certificates\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.219987 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.217219    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a7fc401-27df-4623-b7e4-9f06886fe86a-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ggpsk\" (UID: \"5a7fc401-27df-4623-b7e4-9f06886fe86a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"
Sep 09 10:30:07.219987 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.217294    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe4c67ed-5d54-4d1c-84b4-febd3326c935-installation-pull-secrets\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.219987 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.217608    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe4c67ed-5d54-4d1c-84b4-febd3326c935-trusted-ca\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.220474 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.220452    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe4c67ed-5d54-4d1c-84b4-febd3326c935-image-registry-private-configuration\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.220556 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.220537    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-stats-auth\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.220718 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.220703    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-default-certificate\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.221374 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.221358    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a7fc401-27df-4623-b7e4-9f06886fe86a-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ggpsk\" (UID: \"5a7fc401-27df-4623-b7e4-9f06886fe86a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"
Sep 09 10:30:07.225011 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.224967    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjgv8\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-kube-api-access-bjgv8\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.230547 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.230526    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lcv2\" (UniqueName: \"kubernetes.io/projected/c824a8cb-67bf-4ee7-bc76-919452663ea1-kube-api-access-7lcv2\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.231151 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.231135    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2q2d\" (UniqueName: \"kubernetes.io/projected/5a7fc401-27df-4623-b7e4-9f06886fe86a-kube-api-access-z2q2d\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ggpsk\" (UID: \"5a7fc401-27df-4623-b7e4-9f06886fe86a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"
Sep 09 10:30:07.233324 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.233302    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-bound-sa-token\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.290987 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.290957    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:07.295599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.295584    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"
Sep 09 10:30:07.317340 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.317304    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e16b6c9f-7090-4e95-b0eb-0b07907d992f-serving-cert\") pod \"console-operator-85b454874b-67lpw\" (UID: \"e16b6c9f-7090-4e95-b0eb-0b07907d992f\") " pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.317478 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.317368    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e16b6c9f-7090-4e95-b0eb-0b07907d992f-trusted-ca\") pod \"console-operator-85b454874b-67lpw\" (UID: \"e16b6c9f-7090-4e95-b0eb-0b07907d992f\") " pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.317478 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.317463    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55j2l\" (UniqueName: \"kubernetes.io/projected/e16b6c9f-7090-4e95-b0eb-0b07907d992f-kube-api-access-55j2l\") pod \"console-operator-85b454874b-67lpw\" (UID: \"e16b6c9f-7090-4e95-b0eb-0b07907d992f\") " pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.317587 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.317523    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e16b6c9f-7090-4e95-b0eb-0b07907d992f-config\") pod \"console-operator-85b454874b-67lpw\" (UID: \"e16b6c9f-7090-4e95-b0eb-0b07907d992f\") " pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.318243 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.318220    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e16b6c9f-7090-4e95-b0eb-0b07907d992f-trusted-ca\") pod \"console-operator-85b454874b-67lpw\" (UID: \"e16b6c9f-7090-4e95-b0eb-0b07907d992f\") " pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.318243 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.318231    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e16b6c9f-7090-4e95-b0eb-0b07907d992f-config\") pod \"console-operator-85b454874b-67lpw\" (UID: \"e16b6c9f-7090-4e95-b0eb-0b07907d992f\") " pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.319585 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.319567    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e16b6c9f-7090-4e95-b0eb-0b07907d992f-serving-cert\") pod \"console-operator-85b454874b-67lpw\" (UID: \"e16b6c9f-7090-4e95-b0eb-0b07907d992f\") " pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.326720 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.326704    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55j2l\" (UniqueName: \"kubernetes.io/projected/e16b6c9f-7090-4e95-b0eb-0b07907d992f-kube-api-access-55j2l\") pod \"console-operator-85b454874b-67lpw\" (UID: \"e16b6c9f-7090-4e95-b0eb-0b07907d992f\") " pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.372369 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.372344    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:07.389581 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.389553    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:07.534141 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.534107    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8" event={"ID":"a447e513-d9f3-4354-bcf8-76282e8cd872","Type":"ContainerStarted","Data":"474926eb36bd35137ad222ec05c641a76d5dd7f1f18da8d98da54cb9aca783af"}
Sep 09 10:30:07.534453 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.534434    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:30:07.535102 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.535085    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565d575bbb-b7ds8"
Sep 09 10:30:07.574174 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:30:07.574154    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7fc401_27df_4623_b7e4_9f06886fe86a.slice/crio-261192fd5d6f2e093940f629c0a84eb2e65aec1d29967c6bcad35401700dd55a WatchSource:0}: Error finding container 261192fd5d6f2e093940f629c0a84eb2e65aec1d29967c6bcad35401700dd55a: Status 404 returned error can't find the container with id 261192fd5d6f2e093940f629c0a84eb2e65aec1d29967c6bcad35401700dd55a
Sep 09 10:30:07.580717 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.580696    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk"]
Sep 09 10:30:07.638180 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.638157    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-67lpw"]
Sep 09 10:30:07.640541 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:30:07.640515    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16b6c9f_7090_4e95_b0eb_0b07907d992f.slice/crio-79fc495bac959b97a1149cb72af54e339014aca84610dd3ee1d5022200e1a3f3 WatchSource:0}: Error finding container 79fc495bac959b97a1149cb72af54e339014aca84610dd3ee1d5022200e1a3f3: Status 404 returned error can't find the container with id 79fc495bac959b97a1149cb72af54e339014aca84610dd3ee1d5022200e1a3f3
Sep 09 10:30:07.721705 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.721664    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:07.721705 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.721707    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.721924 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:07.721724    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:07.721924 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:07.721845    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 10:30:07.721924 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:07.721850    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:30:07.721924 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:07.721868    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb4d46669-6v69v: secret "image-registry-tls" not found
Sep 09 10:30:07.721924 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:07.721872    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle podName:c824a8cb-67bf-4ee7-bc76-919452663ea1 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:08.721855549 +0000 UTC m=+248.385494504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle") pod "router-default-75db9dfd44-b2ptj" (UID: "c824a8cb-67bf-4ee7-bc76-919452663ea1") : configmap references non-existent config key: service-ca.crt
Sep 09 10:30:07.721924 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:07.721894    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs podName:c824a8cb-67bf-4ee7-bc76-919452663ea1 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:08.721886996 +0000 UTC m=+248.385525948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs") pod "router-default-75db9dfd44-b2ptj" (UID: "c824a8cb-67bf-4ee7-bc76-919452663ea1") : secret "router-metrics-certs-default" not found
Sep 09 10:30:07.721924 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:07.721904    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls podName:fe4c67ed-5d54-4d1c-84b4-febd3326c935 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:08.721898673 +0000 UTC m=+248.385537624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls") pod "image-registry-fb4d46669-6v69v" (UID: "fe4c67ed-5d54-4d1c-84b4-febd3326c935") : secret "image-registry-tls" not found
Sep 09 10:30:08.539636 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:08.539594    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk" event={"ID":"5a7fc401-27df-4623-b7e4-9f06886fe86a","Type":"ContainerStarted","Data":"261192fd5d6f2e093940f629c0a84eb2e65aec1d29967c6bcad35401700dd55a"}
Sep 09 10:30:08.541473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:08.541412    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-67lpw" event={"ID":"e16b6c9f-7090-4e95-b0eb-0b07907d992f","Type":"ContainerStarted","Data":"79fc495bac959b97a1149cb72af54e339014aca84610dd3ee1d5022200e1a3f3"}
Sep 09 10:30:08.732216 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:08.732174    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:08.732403 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:08.732232    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:08.732403 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:08.732263    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:08.732403 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:08.732346    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:30:08.732403 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:08.732369    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb4d46669-6v69v: secret "image-registry-tls" not found
Sep 09 10:30:08.732614 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:08.732413    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 10:30:08.732614 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:08.732431    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle podName:c824a8cb-67bf-4ee7-bc76-919452663ea1 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:10.73240943 +0000 UTC m=+250.396048400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle") pod "router-default-75db9dfd44-b2ptj" (UID: "c824a8cb-67bf-4ee7-bc76-919452663ea1") : configmap references non-existent config key: service-ca.crt
Sep 09 10:30:08.732614 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:08.732457    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls podName:fe4c67ed-5d54-4d1c-84b4-febd3326c935 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:10.732445501 +0000 UTC m=+250.396084485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls") pod "image-registry-fb4d46669-6v69v" (UID: "fe4c67ed-5d54-4d1c-84b4-febd3326c935") : secret "image-registry-tls" not found
Sep 09 10:30:08.732614 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:08.732476    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs podName:c824a8cb-67bf-4ee7-bc76-919452663ea1 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:10.732466642 +0000 UTC m=+250.396105598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs") pod "router-default-75db9dfd44-b2ptj" (UID: "c824a8cb-67bf-4ee7-bc76-919452663ea1") : secret "router-metrics-certs-default" not found
Sep 09 10:30:09.295004 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:09.294914    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:09.355059 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:09.355026    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:10.549298 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:10.549262    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk" event={"ID":"5a7fc401-27df-4623-b7e4-9f06886fe86a","Type":"ContainerStarted","Data":"1373a4c05aebd586bbb9a705f6f5f566c39f2327e86008fce2345d0e56645658"}
Sep 09 10:30:10.550809 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:10.550792    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/0.log"
Sep 09 10:30:10.550917 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:10.550825    2570 generic.go:356] "Generic (PLEG): container finished" podID="e16b6c9f-7090-4e95-b0eb-0b07907d992f" containerID="b2ed2062d4a590f3ea99c0059d66efe03ae96a2a12ed5c96421fa530e8181d56" exitCode=255
Sep 09 10:30:10.550917 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:10.550862    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-67lpw" event={"ID":"e16b6c9f-7090-4e95-b0eb-0b07907d992f","Type":"ContainerDied","Data":"b2ed2062d4a590f3ea99c0059d66efe03ae96a2a12ed5c96421fa530e8181d56"}
Sep 09 10:30:10.551062 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:10.551047    2570 scope.go:117] "RemoveContainer" containerID="b2ed2062d4a590f3ea99c0059d66efe03ae96a2a12ed5c96421fa530e8181d56"
Sep 09 10:30:10.566198 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:10.566158    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk" podStartSLOduration=2.010563042 podStartE2EDuration="4.566143403s" totalImagesPullingTime="2.555580361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:10.565260241 +0000 UTC m=+250.228899215" watchObservedRunningTime="2026-09-09 10:30:10.566143403 +0000 UTC m=+250.229782377"
Sep 09 10:30:10.747394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:10.747365    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:10.747550 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:10.747401    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:10.747550 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:10.747419    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:10.747550 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:10.747509    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 10:30:10.747550 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:10.747533    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle podName:c824a8cb-67bf-4ee7-bc76-919452663ea1 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:14.747512261 +0000 UTC m=+254.411151215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle") pod "router-default-75db9dfd44-b2ptj" (UID: "c824a8cb-67bf-4ee7-bc76-919452663ea1") : configmap references non-existent config key: service-ca.crt
Sep 09 10:30:10.747550 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:10.747542    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:30:10.747821 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:10.747561    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb4d46669-6v69v: secret "image-registry-tls" not found
Sep 09 10:30:10.747821 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:10.747602    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs podName:c824a8cb-67bf-4ee7-bc76-919452663ea1 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:14.747595757 +0000 UTC m=+254.411234709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs") pod "router-default-75db9dfd44-b2ptj" (UID: "c824a8cb-67bf-4ee7-bc76-919452663ea1") : secret "router-metrics-certs-default" not found
Sep 09 10:30:10.751138 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:10.747976    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls podName:fe4c67ed-5d54-4d1c-84b4-febd3326c935 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:14.747949292 +0000 UTC m=+254.411588262 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls") pod "image-registry-fb4d46669-6v69v" (UID: "fe4c67ed-5d54-4d1c-84b4-febd3326c935") : secret "image-registry-tls" not found
Sep 09 10:30:11.492342 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.492308    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qddgs"]
Sep 09 10:30:11.495380 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.495363    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.498050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.498028    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89f9s\""
Sep 09 10:30:11.498483 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.498468    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 10:30:11.499188 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.499169    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 10:30:11.499280 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.499175    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 10:30:11.499280 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.499177    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 10:30:11.500104 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.500084    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:11.508482 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.508462    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qddgs"]
Sep 09 10:30:11.555897 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.555872    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:30:11.556415 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.556394    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/0.log"
Sep 09 10:30:11.556491 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.556439    2570 generic.go:356] "Generic (PLEG): container finished" podID="e16b6c9f-7090-4e95-b0eb-0b07907d992f" containerID="0688dd26b1d13385557cca106e6e1f9a170b67d1ebee93b9ecd19a3818ea873e" exitCode=255
Sep 09 10:30:11.556553 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.556531    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-67lpw" event={"ID":"e16b6c9f-7090-4e95-b0eb-0b07907d992f","Type":"ContainerDied","Data":"0688dd26b1d13385557cca106e6e1f9a170b67d1ebee93b9ecd19a3818ea873e"}
Sep 09 10:30:11.556598 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.556586    2570 scope.go:117] "RemoveContainer" containerID="b2ed2062d4a590f3ea99c0059d66efe03ae96a2a12ed5c96421fa530e8181d56"
Sep 09 10:30:11.556867 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.556852    2570 scope.go:117] "RemoveContainer" containerID="0688dd26b1d13385557cca106e6e1f9a170b67d1ebee93b9ecd19a3818ea873e"
Sep 09 10:30:11.557036 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:11.557018    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-67lpw_openshift-console-operator(e16b6c9f-7090-4e95-b0eb-0b07907d992f)\"" pod="openshift-console-operator/console-operator-85b454874b-67lpw" podUID="e16b6c9f-7090-4e95-b0eb-0b07907d992f"
Sep 09 10:30:11.569530 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.569513    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:11.654753 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.654717    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs9q6\" (UniqueName: \"kubernetes.io/projected/133066fe-322b-44f6-85f7-3e6f43840330-kube-api-access-vs9q6\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.654954 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.654818    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/133066fe-322b-44f6-85f7-3e6f43840330-crio-socket\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.654954 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.654874    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.654954 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.654901    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.655236 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.655209    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/133066fe-322b-44f6-85f7-3e6f43840330-data-volume\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.755972 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.755889    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.755972 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.755933    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.756187 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.756010    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/133066fe-322b-44f6-85f7-3e6f43840330-data-volume\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.756187 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:11.756046    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:11.756187 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.756074    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs9q6\" (UniqueName: \"kubernetes.io/projected/133066fe-322b-44f6-85f7-3e6f43840330-kube-api-access-vs9q6\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.756187 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:11.756087    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 10:30:11.756187 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:11.756119    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls podName:133066fe-322b-44f6-85f7-3e6f43840330 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:12.256099767 +0000 UTC m=+251.919738725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qddgs" (UID: "133066fe-322b-44f6-85f7-3e6f43840330") : secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:11.756187 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.756168    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/133066fe-322b-44f6-85f7-3e6f43840330-crio-socket\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.756422 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:11.756217    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm podName:133066fe-322b-44f6-85f7-3e6f43840330 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:12.256199144 +0000 UTC m=+251.919838095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qddgs" (UID: "133066fe-322b-44f6-85f7-3e6f43840330") : configmap "kube-rbac-proxy" not found
Sep 09 10:30:11.756422 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.756273    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/133066fe-322b-44f6-85f7-3e6f43840330-crio-socket\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.756422 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.756389    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/133066fe-322b-44f6-85f7-3e6f43840330-data-volume\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.769757 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.769731    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs9q6\" (UniqueName: \"kubernetes.io/projected/133066fe-322b-44f6-85f7-3e6f43840330-kube-api-access-vs9q6\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:11.776119 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:11.776075    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:12.261266 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:12.261236    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:12.261414 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:12.261274    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:12.261414 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:12.261402    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:12.261484 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:12.261439    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 10:30:12.261484 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:12.261480    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls podName:133066fe-322b-44f6-85f7-3e6f43840330 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:13.261449883 +0000 UTC m=+252.925088855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qddgs" (UID: "133066fe-322b-44f6-85f7-3e6f43840330") : secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:12.261557 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:12.261495    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm podName:133066fe-322b-44f6-85f7-3e6f43840330 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:13.261488956 +0000 UTC m=+252.925127908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qddgs" (UID: "133066fe-322b-44f6-85f7-3e6f43840330") : configmap "kube-rbac-proxy" not found
Sep 09 10:30:12.561145 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:12.561076    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:30:12.561485 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:12.561393    2570 scope.go:117] "RemoveContainer" containerID="0688dd26b1d13385557cca106e6e1f9a170b67d1ebee93b9ecd19a3818ea873e"
Sep 09 10:30:12.561555 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:12.561537    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-67lpw_openshift-console-operator(e16b6c9f-7090-4e95-b0eb-0b07907d992f)\"" pod="openshift-console-operator/console-operator-85b454874b-67lpw" podUID="e16b6c9f-7090-4e95-b0eb-0b07907d992f"
Sep 09 10:30:12.664213 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:12.664174    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:30:12.664406 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:12.664316    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 10:30:12.664406 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:12.664392    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs podName:6969769c-d30e-4cbb-afc5-9df38c741545 nodeName:}" failed. No retries permitted until 2026-09-09 10:32:14.664372543 +0000 UTC m=+374.328011494 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs") pod "network-metrics-daemon-llgq5" (UID: "6969769c-d30e-4cbb-afc5-9df38c741545") : secret "metrics-daemon-secret" not found
Sep 09 10:30:12.889597 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:12.889512    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:12.989994 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:12.989959    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:13.269938 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:13.269908    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:13.270092 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:13.269946    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:13.270092 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:13.270053    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 10:30:13.270092 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:13.270065    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:13.270192 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:13.270105    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm podName:133066fe-322b-44f6-85f7-3e6f43840330 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:15.270091398 +0000 UTC m=+254.933730363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qddgs" (UID: "133066fe-322b-44f6-85f7-3e6f43840330") : configmap "kube-rbac-proxy" not found
Sep 09 10:30:13.270192 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:13.270119    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls podName:133066fe-322b-44f6-85f7-3e6f43840330 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:15.270113235 +0000 UTC m=+254.933752186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qddgs" (UID: "133066fe-322b-44f6-85f7-3e6f43840330") : secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:13.438360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:13.438337    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dh892_935d141b-8ba9-42a3-9a46-91f75e24a0bf/dns-node-resolver/0.log"
Sep 09 10:30:14.238341 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:14.238313    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5j47k_55711478-e733-4bdb-a947-0b6f37cc3f75/node-ca/0.log"
Sep 09 10:30:14.784631 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:14.784583    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:14.784631 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:14.784636    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:14.784886 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:14.784659    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:14.784886 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:14.784758    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:30:14.784886 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:14.784816    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb4d46669-6v69v: secret "image-registry-tls" not found
Sep 09 10:30:14.784886 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:14.784846    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle podName:c824a8cb-67bf-4ee7-bc76-919452663ea1 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:22.784826911 +0000 UTC m=+262.448465882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle") pod "router-default-75db9dfd44-b2ptj" (UID: "c824a8cb-67bf-4ee7-bc76-919452663ea1") : configmap references non-existent config key: service-ca.crt
Sep 09 10:30:14.784886 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:14.784853    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 10:30:14.784886 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:14.784864    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls podName:fe4c67ed-5d54-4d1c-84b4-febd3326c935 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:22.784857748 +0000 UTC m=+262.448496703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls") pod "image-registry-fb4d46669-6v69v" (UID: "fe4c67ed-5d54-4d1c-84b4-febd3326c935") : secret "image-registry-tls" not found
Sep 09 10:30:14.785084 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:14.784933    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs podName:c824a8cb-67bf-4ee7-bc76-919452663ea1 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:22.784913346 +0000 UTC m=+262.448552302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs") pod "router-default-75db9dfd44-b2ptj" (UID: "c824a8cb-67bf-4ee7-bc76-919452663ea1") : secret "router-metrics-certs-default" not found
Sep 09 10:30:15.038626 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:15.038547    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5llp7_6d95d07c-cf59-43f5-a825-a0744b99eafb/serve-healthcheck-canary/0.log"
Sep 09 10:30:15.282192 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:15.282164    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:15.288388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:15.288344    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:15.288527 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:15.288393    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:15.288527 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:15.288472    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 10:30:15.288657 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:15.288550    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm podName:133066fe-322b-44f6-85f7-3e6f43840330 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:19.288532774 +0000 UTC m=+258.952171746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qddgs" (UID: "133066fe-322b-44f6-85f7-3e6f43840330") : configmap "kube-rbac-proxy" not found
Sep 09 10:30:15.288657 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:15.288474    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:15.288657 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:15.288588    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls podName:133066fe-322b-44f6-85f7-3e6f43840330 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:19.288578255 +0000 UTC m=+258.952217211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qddgs" (UID: "133066fe-322b-44f6-85f7-3e6f43840330") : secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:15.493223 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:15.493192    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:15.718930 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:15.718850    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:15.824161 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:15.824132    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:15.983509 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:15.983434    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:16.080700 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:16.080671    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:16.144578 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:16.144551    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:17.009965 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:17.009935    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:17.206743 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:17.206712    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:17.373099 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:17.373034    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:17.373099 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:17.373090    2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:17.373434 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:17.373422    2570 scope.go:117] "RemoveContainer" containerID="0688dd26b1d13385557cca106e6e1f9a170b67d1ebee93b9ecd19a3818ea873e"
Sep 09 10:30:17.373585 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:17.373568    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-67lpw_openshift-console-operator(e16b6c9f-7090-4e95-b0eb-0b07907d992f)\"" pod="openshift-console-operator/console-operator-85b454874b-67lpw" podUID="e16b6c9f-7090-4e95-b0eb-0b07907d992f"
Sep 09 10:30:18.152227 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:18.152195    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:18.254569 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:18.254538    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:18.354554 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:18.354518    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:18.452679 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:18.452605    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:18.805257 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:18.805230    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:19.004751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:19.004728    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:19.321828 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:19.321798    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:19.321828 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:19.321830    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:19.322216 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:19.321930    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:19.322216 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:19.321955    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 10:30:19.322216 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:19.321985    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls podName:133066fe-322b-44f6-85f7-3e6f43840330 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:27.321970614 +0000 UTC m=+266.985609566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qddgs" (UID: "133066fe-322b-44f6-85f7-3e6f43840330") : secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:19.322216 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:19.322001    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm podName:133066fe-322b-44f6-85f7-3e6f43840330 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:27.321990468 +0000 UTC m=+266.985629439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qddgs" (UID: "133066fe-322b-44f6-85f7-3e6f43840330") : configmap "kube-rbac-proxy" not found
Sep 09 10:30:19.409280 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:19.409246    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:19.603395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:19.603308    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:19.800570 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:19.800543    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:20.011726 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:20.011694    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:20.401282 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:20.401204    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:20.602925 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:20.602895    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:21.009984 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:21.009956    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:21.403042 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:21.402949    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:21.865828 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:21.865794    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:22.200946 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:22.200853    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:22.301749 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:22.301718    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:22.428574 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:22.428547    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:22.802607 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:22.802572    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:22.850429 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:22.850404    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:22.850602 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:22.850446    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:22.850602 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:22.850475    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:22.851058 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:22.851028    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c824a8cb-67bf-4ee7-bc76-919452663ea1-service-ca-bundle\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:22.852831 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:22.852811    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c824a8cb-67bf-4ee7-bc76-919452663ea1-metrics-certs\") pod \"router-default-75db9dfd44-b2ptj\" (UID: \"c824a8cb-67bf-4ee7-bc76-919452663ea1\") " pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:22.852945 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:22.852888    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls\") pod \"image-registry-fb4d46669-6v69v\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") " pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:22.884161 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:22.884138    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:22.891877 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:22.891860    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:23.008686 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:23.008651    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:23.156021 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:23.155892    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75db9dfd44-b2ptj"]
Sep 09 10:30:23.158749 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:30:23.158724    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc824a8cb_67bf_4ee7_bc76_919452663ea1.slice/crio-b9e676f1876c5fc041f405236e0d104b205d2303806d939b3dc0721396e04ada WatchSource:0}: Error finding container b9e676f1876c5fc041f405236e0d104b205d2303806d939b3dc0721396e04ada: Status 404 returned error can't find the container with id b9e676f1876c5fc041f405236e0d104b205d2303806d939b3dc0721396e04ada
Sep 09 10:30:23.176653 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:23.176612    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb4d46669-6v69v"]
Sep 09 10:30:23.186708 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:30:23.186674    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4c67ed_5d54_4d1c_84b4_febd3326c935.slice/crio-c91439c002e414c21005ffd87973a5cfdd4d7fb18ab6c12d6afcfec7e37a6d7d WatchSource:0}: Error finding container c91439c002e414c21005ffd87973a5cfdd4d7fb18ab6c12d6afcfec7e37a6d7d: Status 404 returned error can't find the container with id c91439c002e414c21005ffd87973a5cfdd4d7fb18ab6c12d6afcfec7e37a6d7d
Sep 09 10:30:23.207975 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:23.207951    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:23.597005 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:23.596973    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb4d46669-6v69v" event={"ID":"fe4c67ed-5d54-4d1c-84b4-febd3326c935","Type":"ContainerStarted","Data":"8c3b391f2bd24edd09bd83f64c5bb5a976850725923ae4e44c0064ac6f69f952"}
Sep 09 10:30:23.597445 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:23.597024    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb4d46669-6v69v" event={"ID":"fe4c67ed-5d54-4d1c-84b4-febd3326c935","Type":"ContainerStarted","Data":"c91439c002e414c21005ffd87973a5cfdd4d7fb18ab6c12d6afcfec7e37a6d7d"}
Sep 09 10:30:23.597445 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:23.597132    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:30:23.598468 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:23.598442    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75db9dfd44-b2ptj" event={"ID":"c824a8cb-67bf-4ee7-bc76-919452663ea1","Type":"ContainerStarted","Data":"2e3df2da0bdce069eef55a39a9c29e1034305afeb71a4311899861c6c7028d13"}
Sep 09 10:30:23.598587 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:23.598488    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75db9dfd44-b2ptj" event={"ID":"c824a8cb-67bf-4ee7-bc76-919452663ea1","Type":"ContainerStarted","Data":"b9e676f1876c5fc041f405236e0d104b205d2303806d939b3dc0721396e04ada"}
Sep 09 10:30:23.619508 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:23.619466    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fb4d46669-6v69v" podStartSLOduration=17.619452442 podStartE2EDuration="17.619452442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:23.61856594 +0000 UTC m=+263.282204915" watchObservedRunningTime="2026-09-09 10:30:23.619452442 +0000 UTC m=+263.283091415"
Sep 09 10:30:23.639257 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:23.639212    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-75db9dfd44-b2ptj" podStartSLOduration=17.639197466 podStartE2EDuration="17.639197466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:23.638145264 +0000 UTC m=+263.301784237" watchObservedRunningTime="2026-09-09 10:30:23.639197466 +0000 UTC m=+263.302836440"
Sep 09 10:30:23.885262 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:23.885179    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:23.887691 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:23.887663    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:24.203508 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:24.203431    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:24.406124 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:24.406093    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:24.601705 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:24.601675    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:24.602942 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:24.602922    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-75db9dfd44-b2ptj"
Sep 09 10:30:24.678322 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:24.678297    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:24.810224 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:24.810193    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:25.023120 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:25.023090    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:25.429969 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:25.429883    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:25.531000 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:25.530968    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:25.630885 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:25.630858    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:26.007826 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:26.007794    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:26.207425 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:26.207392    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:26.608165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:26.608137    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:26.809708 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:26.809677    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:27.386721 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:27.386675    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:27.386721 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:27.386719    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:27.387213 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:27.387194    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/133066fe-322b-44f6-85f7-3e6f43840330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:27.389081 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:27.389057    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/133066fe-322b-44f6-85f7-3e6f43840330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qddgs\" (UID: \"133066fe-322b-44f6-85f7-3e6f43840330\") " pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:27.405417 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:27.405390    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qddgs"
Sep 09 10:30:27.677642 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:27.677448    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qddgs"]
Sep 09 10:30:27.680010 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:30:27.679982    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133066fe_322b_44f6_85f7_3e6f43840330.slice/crio-fd2cfbb18047182934f5d847670044edd186dce3c5f839cc9aad4ad6bf103d0d WatchSource:0}: Error finding container fd2cfbb18047182934f5d847670044edd186dce3c5f839cc9aad4ad6bf103d0d: Status 404 returned error can't find the container with id fd2cfbb18047182934f5d847670044edd186dce3c5f839cc9aad4ad6bf103d0d
Sep 09 10:30:27.887590 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:27.887535    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:28.139715 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:28.139682    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:28.615383 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:28.615351    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qddgs" event={"ID":"133066fe-322b-44f6-85f7-3e6f43840330","Type":"ContainerStarted","Data":"05389f64fcff21282fa39e28d13e973316c0009ed83d15301c9e14aa34efcca3"}
Sep 09 10:30:28.615539 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:28.615407    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qddgs" event={"ID":"133066fe-322b-44f6-85f7-3e6f43840330","Type":"ContainerStarted","Data":"d199fb4aff4d0478edc3cfe0eea249ed502ca80cc8aa433d0d88d95e5358b16b"}
Sep 09 10:30:28.615539 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:28.615427    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qddgs" event={"ID":"133066fe-322b-44f6-85f7-3e6f43840330","Type":"ContainerStarted","Data":"fd2cfbb18047182934f5d847670044edd186dce3c5f839cc9aad4ad6bf103d0d"}
Sep 09 10:30:28.834039 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:28.834000    2570 scope.go:117] "RemoveContainer" containerID="0688dd26b1d13385557cca106e6e1f9a170b67d1ebee93b9ecd19a3818ea873e"
Sep 09 10:30:29.620372 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:29.620348    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:30:29.620736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:29.620418    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-67lpw" event={"ID":"e16b6c9f-7090-4e95-b0eb-0b07907d992f","Type":"ContainerStarted","Data":"2f61172910ee3a282fd7bdaf60805205cea37ee628b2923c9b07bc4eb97f2f0f"}
Sep 09 10:30:29.620736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:29.620669    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:29.625238 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:29.625207    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-67lpw"
Sep 09 10:30:29.640866 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:29.640819    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-67lpw" podStartSLOduration=20.150190329 podStartE2EDuration="22.640803384s" totalImagesPullingTime="2.490613055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:29.639037935 +0000 UTC m=+269.302676908" watchObservedRunningTime="2026-09-09 10:30:29.640803384 +0000 UTC m=+269.304442357"
Sep 09 10:30:29.707618 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:29.707582    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:29.780669 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:29.780626    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:29.842340 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:29.842307    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:30.170393 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:30.170363    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:30.225527 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:30.225497    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:30.818058 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:30.815083    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:30.879053 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:30.878987    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:31.026113 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:31.026077    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:31.126226 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:31.126153    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:31.225744 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:31.225719    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:31.379081 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:31.378996    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:31.556637 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:31.556588    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:31.614034 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:31.614005    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:31.628459 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:31.628431    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qddgs" event={"ID":"133066fe-322b-44f6-85f7-3e6f43840330","Type":"ContainerStarted","Data":"6cf6c96e254bf00ede886270c7bcdaefdd256211b72e3140ce53f517393a3402"}
Sep 09 10:30:31.648169 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:31.648084    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qddgs" podStartSLOduration=17.307388698 podStartE2EDuration="20.648071032s" totalImagesPullingTime="3.340682334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:30:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:31.647277273 +0000 UTC m=+271.310916248" watchObservedRunningTime="2026-09-09 10:30:31.648071032 +0000 UTC m=+271.311710042"
Sep 09 10:30:33.005097 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:33.005060    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:33.114272 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:33.114237    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:35.573726 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.573693    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fb4d46669-6v69v"]
Sep 09 10:30:35.610611 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.610582    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-687d48f8b6-jzq2h"]
Sep 09 10:30:35.613910 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.613890    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.629809 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.629785    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-687d48f8b6-jzq2h"]
Sep 09 10:30:35.651973 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.651941    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/720cdc3c-d15f-4788-99e6-ab6c465ff027-image-registry-private-configuration\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.652118 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.651978    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/720cdc3c-d15f-4788-99e6-ab6c465ff027-registry-certificates\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.652118 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.652000    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/720cdc3c-d15f-4788-99e6-ab6c465ff027-installation-pull-secrets\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.652118 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.652028    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-547l4\" (UniqueName: \"kubernetes.io/projected/720cdc3c-d15f-4788-99e6-ab6c465ff027-kube-api-access-547l4\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.652118 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.652075    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/720cdc3c-d15f-4788-99e6-ab6c465ff027-trusted-ca\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.652118 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.652096    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/720cdc3c-d15f-4788-99e6-ab6c465ff027-registry-tls\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.652118 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.652119    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/720cdc3c-d15f-4788-99e6-ab6c465ff027-ca-trust-extracted\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.652367 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.652136    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/720cdc3c-d15f-4788-99e6-ab6c465ff027-bound-sa-token\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.667660 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.667631    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:35.752484 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.752449    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-547l4\" (UniqueName: \"kubernetes.io/projected/720cdc3c-d15f-4788-99e6-ab6c465ff027-kube-api-access-547l4\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.752649 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.752593    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/720cdc3c-d15f-4788-99e6-ab6c465ff027-trusted-ca\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.752649 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.752622    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/720cdc3c-d15f-4788-99e6-ab6c465ff027-registry-tls\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.752734 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.752650    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/720cdc3c-d15f-4788-99e6-ab6c465ff027-ca-trust-extracted\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.752734 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.752678    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/720cdc3c-d15f-4788-99e6-ab6c465ff027-bound-sa-token\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.752866 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.752739    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/720cdc3c-d15f-4788-99e6-ab6c465ff027-image-registry-private-configuration\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.752866 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.752797    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/720cdc3c-d15f-4788-99e6-ab6c465ff027-registry-certificates\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.752866 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.752828    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/720cdc3c-d15f-4788-99e6-ab6c465ff027-installation-pull-secrets\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.753184 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.753159    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/720cdc3c-d15f-4788-99e6-ab6c465ff027-ca-trust-extracted\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.753787 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.753737    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/720cdc3c-d15f-4788-99e6-ab6c465ff027-trusted-ca\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.753889 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.753743    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/720cdc3c-d15f-4788-99e6-ab6c465ff027-registry-certificates\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.755301 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.755274    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/720cdc3c-d15f-4788-99e6-ab6c465ff027-image-registry-private-configuration\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.755460 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.755440    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/720cdc3c-d15f-4788-99e6-ab6c465ff027-installation-pull-secrets\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.755501 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.755470    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/720cdc3c-d15f-4788-99e6-ab6c465ff027-registry-tls\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.765878 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.765855    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-547l4\" (UniqueName: \"kubernetes.io/projected/720cdc3c-d15f-4788-99e6-ab6c465ff027-kube-api-access-547l4\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.766515 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.766495    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/720cdc3c-d15f-4788-99e6-ab6c465ff027-bound-sa-token\") pod \"image-registry-687d48f8b6-jzq2h\" (UID: \"720cdc3c-d15f-4788-99e6-ab6c465ff027\") " pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:35.776296 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.776274    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:35.923075 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:35.922974    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:36.194894 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:36.194830    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-687d48f8b6-jzq2h"]
Sep 09 10:30:36.197249 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:30:36.197226    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720cdc3c_d15f_4788_99e6_ab6c465ff027.slice/crio-9b65f01cb395efe21d35f1df755802d4c23275fe7f3f017fda58a3737bb1ec83 WatchSource:0}: Error finding container 9b65f01cb395efe21d35f1df755802d4c23275fe7f3f017fda58a3737bb1ec83: Status 404 returned error can't find the container with id 9b65f01cb395efe21d35f1df755802d4c23275fe7f3f017fda58a3737bb1ec83
Sep 09 10:30:36.646139 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:36.646098    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h" event={"ID":"720cdc3c-d15f-4788-99e6-ab6c465ff027","Type":"ContainerStarted","Data":"99d17ec7db6ad8a5e08ba1164fdc20caa922efa73eeec786c3292ab52538f627"}
Sep 09 10:30:36.646589 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:36.646155    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h" event={"ID":"720cdc3c-d15f-4788-99e6-ab6c465ff027","Type":"ContainerStarted","Data":"9b65f01cb395efe21d35f1df755802d4c23275fe7f3f017fda58a3737bb1ec83"}
Sep 09 10:30:36.646589 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:36.646220    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:36.665927 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:36.665870    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h" podStartSLOduration=1.665852062 podStartE2EDuration="1.665852062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:36.665542787 +0000 UTC m=+276.329181762" watchObservedRunningTime="2026-09-09 10:30:36.665852062 +0000 UTC m=+276.329491039"
Sep 09 10:30:39.650300 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.650257    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:39.697869 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.697837    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"]
Sep 09 10:30:39.702565 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.702544    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:39.705250 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.705225    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 10:30:39.705576 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.705560    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 10:30:39.706607 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.706590    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 10:30:39.706952 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.706909    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 10:30:39.707026 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.706949    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 10:30:39.707026 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.707006    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8p4t4\""
Sep 09 10:30:39.710580 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.710562    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"]
Sep 09 10:30:39.786480 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.786452    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3923f82-5f6f-4583-87fb-2b640355eafe-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-6mjnt\" (UID: \"e3923f82-5f6f-4583-87fb-2b640355eafe\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:39.786625 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.786484    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxq6\" (UniqueName: \"kubernetes.io/projected/e3923f82-5f6f-4583-87fb-2b640355eafe-kube-api-access-6bxq6\") pod \"prometheus-operator-6f4946db56-6mjnt\" (UID: \"e3923f82-5f6f-4583-87fb-2b640355eafe\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:39.786625 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.786516    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3923f82-5f6f-4583-87fb-2b640355eafe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-6mjnt\" (UID: \"e3923f82-5f6f-4583-87fb-2b640355eafe\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:39.786625 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.786569    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3923f82-5f6f-4583-87fb-2b640355eafe-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-6mjnt\" (UID: \"e3923f82-5f6f-4583-87fb-2b640355eafe\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:39.852804 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.852775    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:39.887082 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.887054    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3923f82-5f6f-4583-87fb-2b640355eafe-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-6mjnt\" (UID: \"e3923f82-5f6f-4583-87fb-2b640355eafe\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:39.887219 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.887089    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxq6\" (UniqueName: \"kubernetes.io/projected/e3923f82-5f6f-4583-87fb-2b640355eafe-kube-api-access-6bxq6\") pod \"prometheus-operator-6f4946db56-6mjnt\" (UID: \"e3923f82-5f6f-4583-87fb-2b640355eafe\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:39.887219 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.887127    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3923f82-5f6f-4583-87fb-2b640355eafe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-6mjnt\" (UID: \"e3923f82-5f6f-4583-87fb-2b640355eafe\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:39.887219 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.887154    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3923f82-5f6f-4583-87fb-2b640355eafe-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-6mjnt\" (UID: \"e3923f82-5f6f-4583-87fb-2b640355eafe\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:39.887219 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:39.887197    2570 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 09 10:30:39.887406 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:39.887262    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3923f82-5f6f-4583-87fb-2b640355eafe-prometheus-operator-tls podName:e3923f82-5f6f-4583-87fb-2b640355eafe nodeName:}" failed. No retries permitted until 2026-09-09 10:30:40.387245434 +0000 UTC m=+280.050884411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/e3923f82-5f6f-4583-87fb-2b640355eafe-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-6mjnt" (UID: "e3923f82-5f6f-4583-87fb-2b640355eafe") : secret "prometheus-operator-tls" not found
Sep 09 10:30:39.887882 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.887861    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3923f82-5f6f-4583-87fb-2b640355eafe-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-6mjnt\" (UID: \"e3923f82-5f6f-4583-87fb-2b640355eafe\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:39.889536 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.889519    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3923f82-5f6f-4583-87fb-2b640355eafe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-6mjnt\" (UID: \"e3923f82-5f6f-4583-87fb-2b640355eafe\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:39.896412 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:39.896391    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxq6\" (UniqueName: \"kubernetes.io/projected/e3923f82-5f6f-4583-87fb-2b640355eafe-kube-api-access-6bxq6\") pod \"prometheus-operator-6f4946db56-6mjnt\" (UID: \"e3923f82-5f6f-4583-87fb-2b640355eafe\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:40.164549 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:40.164513    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:40.391069 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:40.391030    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3923f82-5f6f-4583-87fb-2b640355eafe-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-6mjnt\" (UID: \"e3923f82-5f6f-4583-87fb-2b640355eafe\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:40.393316 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:40.393291    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3923f82-5f6f-4583-87fb-2b640355eafe-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-6mjnt\" (UID: \"e3923f82-5f6f-4583-87fb-2b640355eafe\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:40.612031 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:40.611997    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"
Sep 09 10:30:40.882249 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:40.882197    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-6mjnt"]
Sep 09 10:30:40.884458 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:30:40.884423    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3923f82_5f6f_4583_87fb_2b640355eafe.slice/crio-f53b4915ca3883856154adea78b95ead5f6c12ddd76c93953dd76ef2f2452467 WatchSource:0}: Error finding container f53b4915ca3883856154adea78b95ead5f6c12ddd76c93953dd76ef2f2452467: Status 404 returned error can't find the container with id f53b4915ca3883856154adea78b95ead5f6c12ddd76c93953dd76ef2f2452467
Sep 09 10:30:41.319781 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:41.319732    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-75ncn" podUID="1c1c0fa9-ce40-473e-a699-540afd0a9ed9"
Sep 09 10:30:41.664339 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:41.664265    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-75ncn"
Sep 09 10:30:41.664339 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:41.664287    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt" event={"ID":"e3923f82-5f6f-4583-87fb-2b640355eafe","Type":"ContainerStarted","Data":"f53b4915ca3883856154adea78b95ead5f6c12ddd76c93953dd76ef2f2452467"}
Sep 09 10:30:43.443724 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:43.443689    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:43.545456 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:43.545420    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:44.231825 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:44.231724    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:44.403583 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:44.403556    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:44.461312 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:44.461281    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:44.676150 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:44.676110    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt" event={"ID":"e3923f82-5f6f-4583-87fb-2b640355eafe","Type":"ContainerStarted","Data":"82c7dcffb92c1b2164654898612f4831a0b6d8a50b9cbbea479626badb3aed98"}
Sep 09 10:30:44.676318 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:44.676171    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt" event={"ID":"e3923f82-5f6f-4583-87fb-2b640355eafe","Type":"ContainerStarted","Data":"3455db9ccf41e11b3ffa446b2126d869c0998054752b1dce45fd35c1c357bb13"}
Sep 09 10:30:45.232739 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:45.232701    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:30:45.235001 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:45.234975    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c1c0fa9-ce40-473e-a699-540afd0a9ed9-metrics-tls\") pod \"dns-default-75ncn\" (UID: \"1c1c0fa9-ce40-473e-a699-540afd0a9ed9\") " pod="openshift-dns/dns-default-75ncn"
Sep 09 10:30:45.268147 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:45.268123    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-272r8\""
Sep 09 10:30:45.275615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:45.275598    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-75ncn"
Sep 09 10:30:45.542491 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:45.542421    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-6mjnt" podStartSLOduration=3.488624104 podStartE2EDuration="6.542400369s" totalImagesPullingTime="3.053776265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:44.693323252 +0000 UTC m=+284.356962226" watchObservedRunningTime="2026-09-09 10:30:45.542400369 +0000 UTC m=+285.206039344"
Sep 09 10:30:45.543626 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:45.543601    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-75ncn"]
Sep 09 10:30:45.547521 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:30:45.547496    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1c0fa9_ce40_473e_a699_540afd0a9ed9.slice/crio-11b84c34f03d47d00f48a0aa2482c12057cd3fd83cbccc0feff741be4d41a547 WatchSource:0}: Error finding container 11b84c34f03d47d00f48a0aa2482c12057cd3fd83cbccc0feff741be4d41a547: Status 404 returned error can't find the container with id 11b84c34f03d47d00f48a0aa2482c12057cd3fd83cbccc0feff741be4d41a547
Sep 09 10:30:45.579264 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:45.579234    2570 patch_prober.go:28] interesting pod/image-registry-fb4d46669-6v69v container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 10:30:45.579365 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:45.579279    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-fb4d46669-6v69v" podUID="fe4c67ed-5d54-4d1c-84b4-febd3326c935" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 10:30:45.680070 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:45.680035    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-75ncn" event={"ID":"1c1c0fa9-ce40-473e-a699-540afd0a9ed9","Type":"ContainerStarted","Data":"11b84c34f03d47d00f48a0aa2482c12057cd3fd83cbccc0feff741be4d41a547"}
Sep 09 10:30:46.189782 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.189737    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5769b56bcd-h9krl"]
Sep 09 10:30:46.192915 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.192890    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.195931 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.195907    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 10:30:46.197157 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.197032    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 10:30:46.197157 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.197036    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 10:30:46.197904 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.197580    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 10:30:46.197904 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.197631    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 10:30:46.198045 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.197957    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 10:30:46.198093 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.198066    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dchvh\""
Sep 09 10:30:46.198293 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.198160    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 10:30:46.206198 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.206167    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5769b56bcd-h9krl"]
Sep 09 10:30:46.206338 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.206319    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 10:30:46.240349 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.240322    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-oauth-serving-cert\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.240490 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.240368    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bf4831f-45ff-4feb-9aad-9237fae12648-console-serving-cert\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.240490 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.240394    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-console-config\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.240490 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.240458    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-service-ca\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.240490 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.240485    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-trusted-ca-bundle\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.240692 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.240589    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw8ml\" (UniqueName: \"kubernetes.io/projected/0bf4831f-45ff-4feb-9aad-9237fae12648-kube-api-access-mw8ml\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.240692 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.240631    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bf4831f-45ff-4feb-9aad-9237fae12648-console-oauth-config\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.341780 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.341688    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-oauth-serving-cert\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.341947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.341812    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bf4831f-45ff-4feb-9aad-9237fae12648-console-serving-cert\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.341947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.341831    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-console-config\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.341947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.341926    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-service-ca\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.342091 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.341955    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-trusted-ca-bundle\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.342091 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.342057    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw8ml\" (UniqueName: \"kubernetes.io/projected/0bf4831f-45ff-4feb-9aad-9237fae12648-kube-api-access-mw8ml\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.342171 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.342116    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bf4831f-45ff-4feb-9aad-9237fae12648-console-oauth-config\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.342638 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.342595    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-console-config\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.342638 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.342621    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-oauth-serving-cert\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.342985 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.342965    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-service-ca\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.343203 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.343182    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-trusted-ca-bundle\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.344841 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.344818    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bf4831f-45ff-4feb-9aad-9237fae12648-console-oauth-config\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.345045 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.345025    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bf4831f-45ff-4feb-9aad-9237fae12648-console-serving-cert\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.352004 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.351984    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw8ml\" (UniqueName: \"kubernetes.io/projected/0bf4831f-45ff-4feb-9aad-9237fae12648-kube-api-access-mw8ml\") pod \"console-5769b56bcd-h9krl\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") " pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.509820 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.509780    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:46.796613 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.796494    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:46.797527 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:46.797385    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5769b56bcd-h9krl"]
Sep 09 10:30:47.027882 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.027821    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:47.071367 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.071284    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tz5nl"]
Sep 09 10:30:47.076275 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.076248    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.079995 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.079968    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 10:30:47.080468 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.080447    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 10:30:47.080595 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.080512    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x8vlp\""
Sep 09 10:30:47.081634 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.081614    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 10:30:47.114642 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:30:47.114613    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf4831f_45ff_4feb_9aad_9237fae12648.slice/crio-ef8182d67fcc404ca4aa948fed9951f877d55e1e9e0ec39a6621140841fd91d7 WatchSource:0}: Error finding container ef8182d67fcc404ca4aa948fed9951f877d55e1e9e0ec39a6621140841fd91d7: Status 404 returned error can't find the container with id ef8182d67fcc404ca4aa948fed9951f877d55e1e9e0ec39a6621140841fd91d7
Sep 09 10:30:47.143399 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.143375    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:47.150396 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.150374    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d154fb6-5d37-4013-b8e1-da86a5a410b6-metrics-client-ca\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.150500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.150407    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d154fb6-5d37-4013-b8e1-da86a5a410b6-root\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.150500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.150433    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mflnl\" (UniqueName: \"kubernetes.io/projected/6d154fb6-5d37-4013-b8e1-da86a5a410b6-kube-api-access-mflnl\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.150500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.150451    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-wtmp\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.150500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.150493    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-textfile\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.150650 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.150513    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.150650 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.150559    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d154fb6-5d37-4013-b8e1-da86a5a410b6-sys\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.150650 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.150577    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-tls\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.150650 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.150600    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.251859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.251401    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-tls\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.251859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.251452    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.251859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.251488    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d154fb6-5d37-4013-b8e1-da86a5a410b6-metrics-client-ca\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.251859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.251523    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d154fb6-5d37-4013-b8e1-da86a5a410b6-root\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.251859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.251560    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mflnl\" (UniqueName: \"kubernetes.io/projected/6d154fb6-5d37-4013-b8e1-da86a5a410b6-kube-api-access-mflnl\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.251859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.251588    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-wtmp\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.251859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.251607    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-textfile\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.251859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.251631    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.251859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.251676    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d154fb6-5d37-4013-b8e1-da86a5a410b6-sys\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.251859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.251778    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d154fb6-5d37-4013-b8e1-da86a5a410b6-sys\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.252378 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.252101    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-textfile\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.252378 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.252164    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d154fb6-5d37-4013-b8e1-da86a5a410b6-root\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.252378 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.252329    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-wtmp\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.252711 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.252651    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.252823 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.252717    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d154fb6-5d37-4013-b8e1-da86a5a410b6-metrics-client-ca\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.254035 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.254011    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-tls\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.254399 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.254382    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d154fb6-5d37-4013-b8e1-da86a5a410b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.264297 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.264276    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mflnl\" (UniqueName: \"kubernetes.io/projected/6d154fb6-5d37-4013-b8e1-da86a5a410b6-kube-api-access-mflnl\") pod \"node-exporter-tz5nl\" (UID: \"6d154fb6-5d37-4013-b8e1-da86a5a410b6\") " pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.295620 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.295587    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:47.388520 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.388493    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tz5nl"
Sep 09 10:30:47.395323 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.395292    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:47.398520 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:30:47.398495    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d154fb6_5d37_4013_b8e1_da86a5a410b6.slice/crio-0e295c1c38dff18e2b64c061b8987bc042ed9a743c5f51e1ec5235ae53fc939c WatchSource:0}: Error finding container 0e295c1c38dff18e2b64c061b8987bc042ed9a743c5f51e1ec5235ae53fc939c: Status 404 returned error can't find the container with id 0e295c1c38dff18e2b64c061b8987bc042ed9a743c5f51e1ec5235ae53fc939c
Sep 09 10:30:47.548437 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.548403    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:47.690462 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.690376    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-75ncn" event={"ID":"1c1c0fa9-ce40-473e-a699-540afd0a9ed9","Type":"ContainerStarted","Data":"8a1fcbbbd256cd33a86c613bb9c4d06b001fc7b5bba2d0cfc872ca317a00ce51"}
Sep 09 10:30:47.690462 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.690441    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-75ncn" event={"ID":"1c1c0fa9-ce40-473e-a699-540afd0a9ed9","Type":"ContainerStarted","Data":"9b949251e1f4fbb976c38ce7e36508e7c993a8d7acdd4ba1d6157f47bf06b548"}
Sep 09 10:30:47.690674 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.690497    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-75ncn"
Sep 09 10:30:47.691868 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.691836    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tz5nl" event={"ID":"6d154fb6-5d37-4013-b8e1-da86a5a410b6","Type":"ContainerStarted","Data":"0e295c1c38dff18e2b64c061b8987bc042ed9a743c5f51e1ec5235ae53fc939c"}
Sep 09 10:30:47.693217 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.693194    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5769b56bcd-h9krl" event={"ID":"0bf4831f-45ff-4feb-9aad-9237fae12648","Type":"ContainerStarted","Data":"ef8182d67fcc404ca4aa948fed9951f877d55e1e9e0ec39a6621140841fd91d7"}
Sep 09 10:30:47.710389 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.710339    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-75ncn" podStartSLOduration=251.0993628 podStartE2EDuration="4m12.7103227s" totalImagesPullingTime="1.6109599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:47.708409682 +0000 UTC m=+287.372048648" watchObservedRunningTime="2026-09-09 10:30:47.7103227 +0000 UTC m=+287.373961675"
Sep 09 10:30:47.808612 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.808578    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:47.909925 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:47.909892    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:48.054612 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:48.054580    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:48.152680 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:48.152640    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:48.491064 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:48.491036    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:48.697684 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:48.697610    2570 generic.go:356] "Generic (PLEG): container finished" podID="6d154fb6-5d37-4013-b8e1-da86a5a410b6" containerID="d669064b2cef9f4b8b73c45d015588010b830c16d28a61765a32c67cd20d1b65" exitCode=0
Sep 09 10:30:48.697896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:48.697689    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tz5nl" event={"ID":"6d154fb6-5d37-4013-b8e1-da86a5a410b6","Type":"ContainerDied","Data":"d669064b2cef9f4b8b73c45d015588010b830c16d28a61765a32c67cd20d1b65"}
Sep 09 10:30:50.707349 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:50.707316    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tz5nl" event={"ID":"6d154fb6-5d37-4013-b8e1-da86a5a410b6","Type":"ContainerStarted","Data":"596074d1eefa74f759ec0bb62b7030c6655ca1a2c155b296fa7c594a6b5227ec"}
Sep 09 10:30:50.707781 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:50.707367    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tz5nl" event={"ID":"6d154fb6-5d37-4013-b8e1-da86a5a410b6","Type":"ContainerStarted","Data":"6a3357ee71bb39fbc1ae44025bf2996a333f9dd24a1192340478bf1969cd43ff"}
Sep 09 10:30:50.708925 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:50.708903    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5769b56bcd-h9krl" event={"ID":"0bf4831f-45ff-4feb-9aad-9237fae12648","Type":"ContainerStarted","Data":"b637d9982fe9c29ab49d5bee00cc6141d94b9ed8ef877ac7bd1965ecd541bc80"}
Sep 09 10:30:50.730564 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:50.730521    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tz5nl" podStartSLOduration=2.650888752 podStartE2EDuration="3.7305088s" totalImagesPullingTime="1.004919561s" totalInitContainerRuntime="74.700487ms" isStatefulPod=true podCreationTimestamp="2026-09-09 10:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:50.730198386 +0000 UTC m=+290.393837361" watchObservedRunningTime="2026-09-09 10:30:50.7305088 +0000 UTC m=+290.394147765"
Sep 09 10:30:50.746459 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:50.746422    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5769b56bcd-h9krl" podStartSLOduration=2.073966365 podStartE2EDuration="4.746409749s" totalImagesPullingTime="2.672443384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:50.746306006 +0000 UTC m=+290.409944992" watchObservedRunningTime="2026-09-09 10:30:50.746409749 +0000 UTC m=+290.410048720"
Sep 09 10:30:51.536687 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:51.536655    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:51.635302 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:51.635270    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:51.826406 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:51.826332    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:51.923780 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:51.923732    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:52.023327 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:52.023300    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:52.834675 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:30:52.834633    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-llgq5" podUID="6969769c-d30e-4cbb-afc5-9df38c741545"
Sep 09 10:30:53.275857 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.275826    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 10:30:53.279474 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.279453    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.285544 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.285520    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 10:30:53.285803 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.285775    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 10:30:53.285902 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.285804    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 10:30:53.285902 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.285876    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 10:30:53.286033 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.286014    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 10:30:53.303426 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.303372    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 10:30:53.303668 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.303647    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 10:30:53.303781 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.303703    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 10:30:53.303861 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.303806    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 10:30:53.303976 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.303949    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 10:30:53.304092 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.304073    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 10:30:53.305960 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.304877    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aqdtqku09ggaj\""
Sep 09 10:30:53.305960 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.305434    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-m4r6v\""
Sep 09 10:30:53.305960 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.305640    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 10:30:53.305960 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.305731    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 10:30:53.312882 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.312861    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 10:30:53.315360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.315339    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 10:30:53.330961 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.330938    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 10:30:53.331346 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.331328    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:53.408174 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408143    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-config\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408174 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408173    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408399 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408198    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408399 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408265    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408399 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408309    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408399 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408335    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx2lh\" (UniqueName: \"kubernetes.io/projected/3afa275a-2875-431c-bb30-10ef6d088c97-kube-api-access-gx2lh\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408399 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408392    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408624 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408419    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408624 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408438    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408624 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408527    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408624 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408556    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408624 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408596    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3afa275a-2875-431c-bb30-10ef6d088c97-config-out\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408624 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408620    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408932 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408655    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3afa275a-2875-431c-bb30-10ef6d088c97-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408932 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408691    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408932 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408716    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408932 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408740    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408932 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408782    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408932 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408813    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.408932 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.408844    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-web-config\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.509733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.509690    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3afa275a-2875-431c-bb30-10ef6d088c97-config-out\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.509942 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.509747    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.509942 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.509795    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3afa275a-2875-431c-bb30-10ef6d088c97-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.509942 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.509830    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.509942 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.509856    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.509942 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.509878    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.509942 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.509903    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.509942 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.509933    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.510292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.509963    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-web-config\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.510292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.509996    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-config\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.510292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.510015    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.510292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.510050    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.510292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.510071    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.510292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.510097    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.510292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.510120    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx2lh\" (UniqueName: \"kubernetes.io/projected/3afa275a-2875-431c-bb30-10ef6d088c97-kube-api-access-gx2lh\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.510292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.510159    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.510292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.510186    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.510292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.510207    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.510292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.510273    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.510292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.510292    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.511000 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.510975    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.512110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.511924    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.512598 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.512571    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.513021 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.513001    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.515793 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.514789    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-config\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.515793 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.513257    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.515793 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.513500    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.515793 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.514460    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.515793 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.514214    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3afa275a-2875-431c-bb30-10ef6d088c97-config-out\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.515793 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.514953    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.515793 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.515201    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.515793 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.515563    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.516258 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.516045    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-web-config\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.516944 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.516901    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.517080 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.517050    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.517382 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.517361    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.517559 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.517537    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3afa275a-2875-431c-bb30-10ef6d088c97-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.517901 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.517883    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.518247 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.518226    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.523276 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.523249    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx2lh\" (UniqueName: \"kubernetes.io/projected/3afa275a-2875-431c-bb30-10ef6d088c97-kube-api-access-gx2lh\") pod \"prometheus-k8s-0\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.590002 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.589916    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:30:53.881385 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:53.881307    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 10:30:53.884408 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:30:53.884381    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afa275a_2875_431c_bb30_10ef6d088c97.slice/crio-07669f218f8e646fcfb9a9fbff9f72b6a232ce64b1ba7122c10d8ebe03949246 WatchSource:0}: Error finding container 07669f218f8e646fcfb9a9fbff9f72b6a232ce64b1ba7122c10d8ebe03949246: Status 404 returned error can't find the container with id 07669f218f8e646fcfb9a9fbff9f72b6a232ce64b1ba7122c10d8ebe03949246
Sep 09 10:30:54.138084 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:54.138000    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:54.238754 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:54.238725    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:54.724651 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:54.724609    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerStarted","Data":"07669f218f8e646fcfb9a9fbff9f72b6a232ce64b1ba7122c10d8ebe03949246"}
Sep 09 10:30:55.579052 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:55.579019    2570 patch_prober.go:28] interesting pod/image-registry-fb4d46669-6v69v container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 10:30:55.579411 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:55.579072    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-fb4d46669-6v69v" podUID="fe4c67ed-5d54-4d1c-84b4-febd3326c935" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 10:30:55.731246 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:55.731214    2570 generic.go:356] "Generic (PLEG): container finished" podID="3afa275a-2875-431c-bb30-10ef6d088c97" containerID="6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650" exitCode=0
Sep 09 10:30:55.731401 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:55.731304    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerDied","Data":"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"}
Sep 09 10:30:55.927210 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:55.927134    2570 patch_prober.go:28] interesting pod/image-registry-687d48f8b6-jzq2h container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 10:30:55.927210 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:55.927184    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h" podUID="720cdc3c-d15f-4788-99e6-ab6c465ff027" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 10:30:56.510391 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:56.510354    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:56.510561 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:56.510435    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:30:56.511971 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:56.511941    2570 patch_prober.go:28] interesting pod/console-5769b56bcd-h9krl container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" start-of-body=
Sep 09 10:30:56.512107 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:56.511993    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5769b56bcd-h9krl" podUID="0bf4831f-45ff-4feb-9aad-9237fae12648" containerName="console" probeResult="failure" output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused"
Sep 09 10:30:57.655343 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:57.655311    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-687d48f8b6-jzq2h"
Sep 09 10:30:57.699397 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:57.699374    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-75ncn"
Sep 09 10:30:57.842671 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:57.842641    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:57.941667 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:57.941584    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:58.044137 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:58.044093    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:30:59.754871 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:59.754833    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerStarted","Data":"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"}
Sep 09 10:30:59.755222 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:30:59.754900    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerStarted","Data":"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"}
Sep 09 10:31:00.600594 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:00.600549    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-fb4d46669-6v69v" podUID="fe4c67ed-5d54-4d1c-84b4-febd3326c935" containerName="registry" containerID="cri-o://8c3b391f2bd24edd09bd83f64c5bb5a976850725923ae4e44c0064ac6f69f952" gracePeriod=30
Sep 09 10:31:00.734500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:00.734467    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:00.971397 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:00.969129    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:31:00.971397 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:00.969154    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:31:00.976120 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:00.976097    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:31:00.976232 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:00.976125    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:31:00.981266 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:00.981248    2570 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 10:31:01.113655 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.113630    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:31:01.290999 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.290964    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-certificates\") pod \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") "
Sep 09 10:31:01.290999 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.291002    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe4c67ed-5d54-4d1c-84b4-febd3326c935-image-registry-private-configuration\") pod \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") "
Sep 09 10:31:01.291221 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.291050    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjgv8\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-kube-api-access-bjgv8\") pod \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") "
Sep 09 10:31:01.291221 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.291084    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe4c67ed-5d54-4d1c-84b4-febd3326c935-ca-trust-extracted\") pod \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") "
Sep 09 10:31:01.291221 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.291122    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe4c67ed-5d54-4d1c-84b4-febd3326c935-trusted-ca\") pod \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") "
Sep 09 10:31:01.291221 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.291186    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe4c67ed-5d54-4d1c-84b4-febd3326c935-installation-pull-secrets\") pod \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") "
Sep 09 10:31:01.291416 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.291237    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls\") pod \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") "
Sep 09 10:31:01.291473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.291436    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fe4c67ed-5d54-4d1c-84b4-febd3326c935" (UID: "fe4c67ed-5d54-4d1c-84b4-febd3326c935"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:31:01.291594 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.291549    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4c67ed-5d54-4d1c-84b4-febd3326c935-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fe4c67ed-5d54-4d1c-84b4-febd3326c935" (UID: "fe4c67ed-5d54-4d1c-84b4-febd3326c935"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:31:01.291734 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.291607    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-bound-sa-token\") pod \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\" (UID: \"fe4c67ed-5d54-4d1c-84b4-febd3326c935\") "
Sep 09 10:31:01.292247 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.292225    2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-certificates\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:01.292338 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.292254    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe4c67ed-5d54-4d1c-84b4-febd3326c935-trusted-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:01.293418 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.293391    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe4c67ed-5d54-4d1c-84b4-febd3326c935-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fe4c67ed-5d54-4d1c-84b4-febd3326c935" (UID: "fe4c67ed-5d54-4d1c-84b4-febd3326c935"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:31:01.293635 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.293608    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe4c67ed-5d54-4d1c-84b4-febd3326c935-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fe4c67ed-5d54-4d1c-84b4-febd3326c935" (UID: "fe4c67ed-5d54-4d1c-84b4-febd3326c935"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:31:01.293737 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.293716    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-kube-api-access-bjgv8" (OuterVolumeSpecName: "kube-api-access-bjgv8") pod "fe4c67ed-5d54-4d1c-84b4-febd3326c935" (UID: "fe4c67ed-5d54-4d1c-84b4-febd3326c935"). InnerVolumeSpecName "kube-api-access-bjgv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:31:01.293822 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.293711    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fe4c67ed-5d54-4d1c-84b4-febd3326c935" (UID: "fe4c67ed-5d54-4d1c-84b4-febd3326c935"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:31:01.294144 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.294117    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fe4c67ed-5d54-4d1c-84b4-febd3326c935" (UID: "fe4c67ed-5d54-4d1c-84b4-febd3326c935"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:31:01.299468 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.299450    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4c67ed-5d54-4d1c-84b4-febd3326c935-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fe4c67ed-5d54-4d1c-84b4-febd3326c935" (UID: "fe4c67ed-5d54-4d1c-84b4-febd3326c935"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:31:01.393340 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.393311    2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe4c67ed-5d54-4d1c-84b4-febd3326c935-ca-trust-extracted\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:01.393340 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.393336    2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe4c67ed-5d54-4d1c-84b4-febd3326c935-installation-pull-secrets\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:01.393340 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.393347    2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-registry-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:01.393556 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.393356    2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-bound-sa-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:01.393556 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.393364    2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe4c67ed-5d54-4d1c-84b4-febd3326c935-image-registry-private-configuration\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:01.393556 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.393373    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjgv8\" (UniqueName: \"kubernetes.io/projected/fe4c67ed-5d54-4d1c-84b4-febd3326c935-kube-api-access-bjgv8\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:01.766119 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.766084    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerStarted","Data":"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"}
Sep 09 10:31:01.766298 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.766189    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerStarted","Data":"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"}
Sep 09 10:31:01.766298 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.766232    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerStarted","Data":"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"}
Sep 09 10:31:01.766298 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.766260    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerStarted","Data":"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"}
Sep 09 10:31:01.767354 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.767329    2570 generic.go:356] "Generic (PLEG): container finished" podID="fe4c67ed-5d54-4d1c-84b4-febd3326c935" containerID="8c3b391f2bd24edd09bd83f64c5bb5a976850725923ae4e44c0064ac6f69f952" exitCode=0
Sep 09 10:31:01.767473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.767363    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb4d46669-6v69v" event={"ID":"fe4c67ed-5d54-4d1c-84b4-febd3326c935","Type":"ContainerDied","Data":"8c3b391f2bd24edd09bd83f64c5bb5a976850725923ae4e44c0064ac6f69f952"}
Sep 09 10:31:01.767473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.767378    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb4d46669-6v69v"
Sep 09 10:31:01.767473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.767399    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb4d46669-6v69v" event={"ID":"fe4c67ed-5d54-4d1c-84b4-febd3326c935","Type":"ContainerDied","Data":"c91439c002e414c21005ffd87973a5cfdd4d7fb18ab6c12d6afcfec7e37a6d7d"}
Sep 09 10:31:01.767473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.767425    2570 scope.go:117] "RemoveContainer" containerID="8c3b391f2bd24edd09bd83f64c5bb5a976850725923ae4e44c0064ac6f69f952"
Sep 09 10:31:01.776515 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.776493    2570 scope.go:117] "RemoveContainer" containerID="8c3b391f2bd24edd09bd83f64c5bb5a976850725923ae4e44c0064ac6f69f952"
Sep 09 10:31:01.776865 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:31:01.776839    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3b391f2bd24edd09bd83f64c5bb5a976850725923ae4e44c0064ac6f69f952\": container with ID starting with 8c3b391f2bd24edd09bd83f64c5bb5a976850725923ae4e44c0064ac6f69f952 not found: ID does not exist" containerID="8c3b391f2bd24edd09bd83f64c5bb5a976850725923ae4e44c0064ac6f69f952"
Sep 09 10:31:01.776985 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.776869    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3b391f2bd24edd09bd83f64c5bb5a976850725923ae4e44c0064ac6f69f952"} err="failed to get container status \"8c3b391f2bd24edd09bd83f64c5bb5a976850725923ae4e44c0064ac6f69f952\": rpc error: code = NotFound desc = could not find container \"8c3b391f2bd24edd09bd83f64c5bb5a976850725923ae4e44c0064ac6f69f952\": container with ID starting with 8c3b391f2bd24edd09bd83f64c5bb5a976850725923ae4e44c0064ac6f69f952 not found: ID does not exist"
Sep 09 10:31:01.798744 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.798701    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.552937828 podStartE2EDuration="8.798688002s" totalImagesPullingTime="6.226611267s" totalInitContainerRuntime="19.138907ms" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:31:01.797425195 +0000 UTC m=+301.461064179" watchObservedRunningTime="2026-09-09 10:31:01.798688002 +0000 UTC m=+301.462326975"
Sep 09 10:31:01.810957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.810935    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fb4d46669-6v69v"]
Sep 09 10:31:01.814921 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:01.814900    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fb4d46669-6v69v"]
Sep 09 10:31:02.305148 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:02.305116    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:02.487909 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:02.487820    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5769b56bcd-h9krl"]
Sep 09 10:31:02.615842 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:02.615811    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:02.715559 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:02.715526    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:02.833442 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:02.833410    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:02.837665 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:02.837640    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe4c67ed-5d54-4d1c-84b4-febd3326c935" path="/var/lib/kubelet/pods/fe4c67ed-5d54-4d1c-84b4-febd3326c935/volumes"
Sep 09 10:31:02.903679 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:02.903648    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:03.590752 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:03.590725    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:31:04.453946 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:04.453914    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:04.529125 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:04.529096    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:06.469440 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:06.469402    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:06.834165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:06.834134    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:31:07.073032 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:07.072988    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:09.065052 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:09.065015    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:09.135948 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:09.135912    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:12.897706 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:12.897674    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:12.998376 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:12.998343    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:18.135760 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:18.135724    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:18.233686 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:18.233649    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:19.189463 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:19.189431    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:19.290229 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:19.290195    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:19.391823 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:19.391792    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:22.478781 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:22.478736    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:22.579503 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:22.579465    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:27.507165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.507122    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5769b56bcd-h9krl" podUID="0bf4831f-45ff-4feb-9aad-9237fae12648" containerName="console" containerID="cri-o://b637d9982fe9c29ab49d5bee00cc6141d94b9ed8ef877ac7bd1965ecd541bc80" gracePeriod=15
Sep 09 10:31:27.753514 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.753488    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:31:27.815347 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.815268    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-service-ca\") pod \"0bf4831f-45ff-4feb-9aad-9237fae12648\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") "
Sep 09 10:31:27.815487 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.815346    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw8ml\" (UniqueName: \"kubernetes.io/projected/0bf4831f-45ff-4feb-9aad-9237fae12648-kube-api-access-mw8ml\") pod \"0bf4831f-45ff-4feb-9aad-9237fae12648\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") "
Sep 09 10:31:27.815487 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.815446    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bf4831f-45ff-4feb-9aad-9237fae12648-console-oauth-config\") pod \"0bf4831f-45ff-4feb-9aad-9237fae12648\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") "
Sep 09 10:31:27.815602 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.815510    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-trusted-ca-bundle\") pod \"0bf4831f-45ff-4feb-9aad-9237fae12648\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") "
Sep 09 10:31:27.815602 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.815570    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-oauth-serving-cert\") pod \"0bf4831f-45ff-4feb-9aad-9237fae12648\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") "
Sep 09 10:31:27.815602 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.815595    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-console-config\") pod \"0bf4831f-45ff-4feb-9aad-9237fae12648\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") "
Sep 09 10:31:27.815742 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.815615    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bf4831f-45ff-4feb-9aad-9237fae12648-console-serving-cert\") pod \"0bf4831f-45ff-4feb-9aad-9237fae12648\" (UID: \"0bf4831f-45ff-4feb-9aad-9237fae12648\") "
Sep 09 10:31:27.815832 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.815806    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-service-ca" (OuterVolumeSpecName: "service-ca") pod "0bf4831f-45ff-4feb-9aad-9237fae12648" (UID: "0bf4831f-45ff-4feb-9aad-9237fae12648"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:31:27.815953 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.815935    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:27.816023 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.815998    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0bf4831f-45ff-4feb-9aad-9237fae12648" (UID: "0bf4831f-45ff-4feb-9aad-9237fae12648"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:31:27.816085 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.816010    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-console-config" (OuterVolumeSpecName: "console-config") pod "0bf4831f-45ff-4feb-9aad-9237fae12648" (UID: "0bf4831f-45ff-4feb-9aad-9237fae12648"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:31:27.816085 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.816019    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0bf4831f-45ff-4feb-9aad-9237fae12648" (UID: "0bf4831f-45ff-4feb-9aad-9237fae12648"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:31:27.817609 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.817579    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf4831f-45ff-4feb-9aad-9237fae12648-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0bf4831f-45ff-4feb-9aad-9237fae12648" (UID: "0bf4831f-45ff-4feb-9aad-9237fae12648"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:31:27.817702 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.817620    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf4831f-45ff-4feb-9aad-9237fae12648-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0bf4831f-45ff-4feb-9aad-9237fae12648" (UID: "0bf4831f-45ff-4feb-9aad-9237fae12648"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:31:27.817702 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.817666    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf4831f-45ff-4feb-9aad-9237fae12648-kube-api-access-mw8ml" (OuterVolumeSpecName: "kube-api-access-mw8ml") pod "0bf4831f-45ff-4feb-9aad-9237fae12648" (UID: "0bf4831f-45ff-4feb-9aad-9237fae12648"). InnerVolumeSpecName "kube-api-access-mw8ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:31:27.864878 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.864846    2570 generic.go:356] "Generic (PLEG): container finished" podID="0bf4831f-45ff-4feb-9aad-9237fae12648" containerID="b637d9982fe9c29ab49d5bee00cc6141d94b9ed8ef877ac7bd1965ecd541bc80" exitCode=0
Sep 09 10:31:27.865015 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.864886    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5769b56bcd-h9krl"
Sep 09 10:31:27.865015 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.864927    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5769b56bcd-h9krl" event={"ID":"0bf4831f-45ff-4feb-9aad-9237fae12648","Type":"ContainerDied","Data":"b637d9982fe9c29ab49d5bee00cc6141d94b9ed8ef877ac7bd1965ecd541bc80"}
Sep 09 10:31:27.865015 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.864980    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5769b56bcd-h9krl" event={"ID":"0bf4831f-45ff-4feb-9aad-9237fae12648","Type":"ContainerDied","Data":"ef8182d67fcc404ca4aa948fed9951f877d55e1e9e0ec39a6621140841fd91d7"}
Sep 09 10:31:27.865015 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.865000    2570 scope.go:117] "RemoveContainer" containerID="b637d9982fe9c29ab49d5bee00cc6141d94b9ed8ef877ac7bd1965ecd541bc80"
Sep 09 10:31:27.873568 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.873546    2570 scope.go:117] "RemoveContainer" containerID="b637d9982fe9c29ab49d5bee00cc6141d94b9ed8ef877ac7bd1965ecd541bc80"
Sep 09 10:31:27.873885 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:31:27.873863    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b637d9982fe9c29ab49d5bee00cc6141d94b9ed8ef877ac7bd1965ecd541bc80\": container with ID starting with b637d9982fe9c29ab49d5bee00cc6141d94b9ed8ef877ac7bd1965ecd541bc80 not found: ID does not exist" containerID="b637d9982fe9c29ab49d5bee00cc6141d94b9ed8ef877ac7bd1965ecd541bc80"
Sep 09 10:31:27.873940 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.873893    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b637d9982fe9c29ab49d5bee00cc6141d94b9ed8ef877ac7bd1965ecd541bc80"} err="failed to get container status \"b637d9982fe9c29ab49d5bee00cc6141d94b9ed8ef877ac7bd1965ecd541bc80\": rpc error: code = NotFound desc = could not find container \"b637d9982fe9c29ab49d5bee00cc6141d94b9ed8ef877ac7bd1965ecd541bc80\": container with ID starting with b637d9982fe9c29ab49d5bee00cc6141d94b9ed8ef877ac7bd1965ecd541bc80 not found: ID does not exist"
Sep 09 10:31:27.886394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.886373    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5769b56bcd-h9krl"]
Sep 09 10:31:27.892197 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.892176    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5769b56bcd-h9krl"]
Sep 09 10:31:27.917391 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.917364    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw8ml\" (UniqueName: \"kubernetes.io/projected/0bf4831f-45ff-4feb-9aad-9237fae12648-kube-api-access-mw8ml\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:27.917391 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.917386    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bf4831f-45ff-4feb-9aad-9237fae12648-console-oauth-config\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:27.917391 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.917394    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-trusted-ca-bundle\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:27.917616 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.917404    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-oauth-serving-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:27.917616 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.917418    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bf4831f-45ff-4feb-9aad-9237fae12648-console-config\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:27.917616 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:27.917430    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bf4831f-45ff-4feb-9aad-9237fae12648-console-serving-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:31:28.837895 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:28.837864    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bf4831f-45ff-4feb-9aad-9237fae12648" path="/var/lib/kubelet/pods/0bf4831f-45ff-4feb-9aad-9237fae12648/volumes"
Sep 09 10:31:40.913411 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:40.913376    2570 generic.go:356] "Generic (PLEG): container finished" podID="5a7fc401-27df-4623-b7e4-9f06886fe86a" containerID="1373a4c05aebd586bbb9a705f6f5f566c39f2327e86008fce2345d0e56645658" exitCode=0
Sep 09 10:31:40.913823 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:40.913448    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk" event={"ID":"5a7fc401-27df-4623-b7e4-9f06886fe86a","Type":"ContainerDied","Data":"1373a4c05aebd586bbb9a705f6f5f566c39f2327e86008fce2345d0e56645658"}
Sep 09 10:31:40.913870 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:40.913822    2570 scope.go:117] "RemoveContainer" containerID="1373a4c05aebd586bbb9a705f6f5f566c39f2327e86008fce2345d0e56645658"
Sep 09 10:31:41.919693 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:41.919658    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ggpsk" event={"ID":"5a7fc401-27df-4623-b7e4-9f06886fe86a","Type":"ContainerStarted","Data":"12e0c66138252d0ef9279cf86b4c18c799dafa25dedbd9c6f690ec2746c29a7c"}
Sep 09 10:31:42.895613 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:42.895574    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:44.600885 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:44.600850    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:31:53.590541 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:53.590447    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:31:53.605660 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:53.605637    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:31:53.978181 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:31:53.978105    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:02.750444 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:02.750407    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:02.850958 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:02.850925    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:06.281558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:06.281522    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:07.173561 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:07.173504    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:11.683975 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.683942    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 10:32:11.684550 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.684364    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="prometheus" containerID="cri-o://3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a" gracePeriod=600
Sep 09 10:32:11.684550 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.684397    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa" gracePeriod=600
Sep 09 10:32:11.684550 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.684398    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="kube-rbac-proxy" containerID="cri-o://3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f" gracePeriod=600
Sep 09 10:32:11.684550 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.684416    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="config-reloader" containerID="cri-o://6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1" gracePeriod=600
Sep 09 10:32:11.684757 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.684604    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="thanos-sidecar" containerID="cri-o://495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc" gracePeriod=600
Sep 09 10:32:11.684757 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.684693    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="kube-rbac-proxy-web" containerID="cri-o://2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d" gracePeriod=600
Sep 09 10:32:11.930018 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.929994    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:11.985054 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.984968    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-serving-certs-ca-bundle\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985054 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985004    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985054 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985025    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-metrics-client-ca\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985054 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985047    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-thanos-prometheus-http-client-file\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985077    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-config\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985097    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-2\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985204    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3afa275a-2875-431c-bb30-10ef6d088c97-tls-assets\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985251    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-0\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985297    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985326    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-kube-rbac-proxy\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985357    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-trusted-ca-bundle\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985392    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-1\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985813 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985400    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:32:11.985813 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985421    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-kubelet-serving-ca-bundle\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985813 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985453    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-tls\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.985813 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985446    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:32:11.985813 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985469    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:32:11.985813 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985693    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:32:11.985813 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985748    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:32:11.986189 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.985903    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:32:11.986971 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.986941    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:32:11.987101 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.986995    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-db\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.987101 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.987061    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-web-config\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.987220 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.987101    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3afa275a-2875-431c-bb30-10ef6d088c97-config-out\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.987220 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.987149    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-metrics-client-certs\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.987220 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.987176    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-grpc-tls\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.987220 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.987200    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx2lh\" (UniqueName: \"kubernetes.io/projected/3afa275a-2875-431c-bb30-10ef6d088c97-kube-api-access-gx2lh\") pod \"3afa275a-2875-431c-bb30-10ef6d088c97\" (UID: \"3afa275a-2875-431c-bb30-10ef6d088c97\") "
Sep 09 10:32:11.987474 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.987454    2570 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:11.987555 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.987479    2570 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-metrics-client-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:11.987555 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.987497    2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:11.987555 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.987509    2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:11.987555 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.987523    2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:11.987555 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.987535    2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:11.987555 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.987548    2570 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afa275a-2875-431c-bb30-10ef6d088c97-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:11.988386 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.988110    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:11.988637 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.988598    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:11.988637 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.988619    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afa275a-2875-431c-bb30-10ef6d088c97-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:32:11.989019 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.988995    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:11.989194 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.989146    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-config" (OuterVolumeSpecName: "config") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:11.989296 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.989257    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:32:11.989296 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.989282    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:11.989831 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.989806    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afa275a-2875-431c-bb30-10ef6d088c97-kube-api-access-gx2lh" (OuterVolumeSpecName: "kube-api-access-gx2lh") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "kube-api-access-gx2lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:32:11.990326 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.990307    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:11.990726 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.990709    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:11.991206 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.991185    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:11.991256 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:11.991199    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afa275a-2875-431c-bb30-10ef6d088c97-config-out" (OuterVolumeSpecName: "config-out") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:32:12.001266 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.001246    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-web-config" (OuterVolumeSpecName: "web-config") pod "3afa275a-2875-431c-bb30-10ef6d088c97" (UID: "3afa275a-2875-431c-bb30-10ef6d088c97"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:12.030315 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030280    2570 generic.go:356] "Generic (PLEG): container finished" podID="3afa275a-2875-431c-bb30-10ef6d088c97" containerID="4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa" exitCode=0
Sep 09 10:32:12.030315 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030311    2570 generic.go:356] "Generic (PLEG): container finished" podID="3afa275a-2875-431c-bb30-10ef6d088c97" containerID="3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f" exitCode=0
Sep 09 10:32:12.030476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030323    2570 generic.go:356] "Generic (PLEG): container finished" podID="3afa275a-2875-431c-bb30-10ef6d088c97" containerID="2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d" exitCode=0
Sep 09 10:32:12.030476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030333    2570 generic.go:356] "Generic (PLEG): container finished" podID="3afa275a-2875-431c-bb30-10ef6d088c97" containerID="495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc" exitCode=0
Sep 09 10:32:12.030476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030342    2570 generic.go:356] "Generic (PLEG): container finished" podID="3afa275a-2875-431c-bb30-10ef6d088c97" containerID="6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1" exitCode=0
Sep 09 10:32:12.030476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030353    2570 generic.go:356] "Generic (PLEG): container finished" podID="3afa275a-2875-431c-bb30-10ef6d088c97" containerID="3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a" exitCode=0
Sep 09 10:32:12.030476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030361    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerDied","Data":"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"}
Sep 09 10:32:12.030476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030383    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.030476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030435    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerDied","Data":"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"}
Sep 09 10:32:12.030700 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030486    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerDied","Data":"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"}
Sep 09 10:32:12.030700 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030504    2570 scope.go:117] "RemoveContainer" containerID="4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"
Sep 09 10:32:12.030700 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030518    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerDied","Data":"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"}
Sep 09 10:32:12.030700 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030545    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerDied","Data":"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"}
Sep 09 10:32:12.030700 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030570    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerDied","Data":"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"}
Sep 09 10:32:12.030700 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.030598    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3afa275a-2875-431c-bb30-10ef6d088c97","Type":"ContainerDied","Data":"07669f218f8e646fcfb9a9fbff9f72b6a232ce64b1ba7122c10d8ebe03949246"}
Sep 09 10:32:12.044182 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.044163    2570 scope.go:117] "RemoveContainer" containerID="3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"
Sep 09 10:32:12.051749 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.051732    2570 scope.go:117] "RemoveContainer" containerID="2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"
Sep 09 10:32:12.058552 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.058529    2570 scope.go:117] "RemoveContainer" containerID="495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"
Sep 09 10:32:12.063997 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.063977    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 10:32:12.065592 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.065574    2570 scope.go:117] "RemoveContainer" containerID="6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"
Sep 09 10:32:12.070320 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.070300    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 10:32:12.072446 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.072430    2570 scope.go:117] "RemoveContainer" containerID="3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"
Sep 09 10:32:12.080508 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.079637    2570 scope.go:117] "RemoveContainer" containerID="6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"
Sep 09 10:32:12.087261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.087243    2570 scope.go:117] "RemoveContainer" containerID="4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"
Sep 09 10:32:12.087517 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:32:12.087494    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa\": container with ID starting with 4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa not found: ID does not exist" containerID="4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"
Sep 09 10:32:12.087568 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.087521    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"} err="failed to get container status \"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa\": rpc error: code = NotFound desc = could not find container \"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa\": container with ID starting with 4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa not found: ID does not exist"
Sep 09 10:32:12.087568 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.087540    2570 scope.go:117] "RemoveContainer" containerID="3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"
Sep 09 10:32:12.087814 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:32:12.087797    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f\": container with ID starting with 3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f not found: ID does not exist" containerID="3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"
Sep 09 10:32:12.087889 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.087819    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"} err="failed to get container status \"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f\": rpc error: code = NotFound desc = could not find container \"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f\": container with ID starting with 3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f not found: ID does not exist"
Sep 09 10:32:12.087889 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.087832    2570 scope.go:117] "RemoveContainer" containerID="2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"
Sep 09 10:32:12.087968 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.087948    2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3afa275a-2875-431c-bb30-10ef6d088c97-prometheus-k8s-db\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:12.088003 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.087967    2570 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-web-config\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:12.088003 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.087982    2570 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3afa275a-2875-431c-bb30-10ef6d088c97-config-out\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:12.088003 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.087990    2570 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-metrics-client-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:12.088003 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.087999    2570 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-grpc-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:12.088126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088010    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx2lh\" (UniqueName: \"kubernetes.io/projected/3afa275a-2875-431c-bb30-10ef6d088c97-kube-api-access-gx2lh\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:12.088126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088025    2570 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:12.088126 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:32:12.088025    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d\": container with ID starting with 2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d not found: ID does not exist" containerID="2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"
Sep 09 10:32:12.088126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088037    2570 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:12.088126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088046    2570 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-config\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:12.088126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088046    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"} err="failed to get container status \"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d\": rpc error: code = NotFound desc = could not find container \"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d\": container with ID starting with 2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d not found: ID does not exist"
Sep 09 10:32:12.088126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088066    2570 scope.go:117] "RemoveContainer" containerID="495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"
Sep 09 10:32:12.088126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088053    2570 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3afa275a-2875-431c-bb30-10ef6d088c97-tls-assets\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:12.088126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088116    2570 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:12.088126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088126    2570 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-kube-rbac-proxy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:12.088449 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088138    2570 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3afa275a-2875-431c-bb30-10ef6d088c97-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:32:12.088449 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:32:12.088281    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc\": container with ID starting with 495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc not found: ID does not exist" containerID="495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"
Sep 09 10:32:12.088449 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088299    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"} err="failed to get container status \"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc\": rpc error: code = NotFound desc = could not find container \"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc\": container with ID starting with 495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc not found: ID does not exist"
Sep 09 10:32:12.088449 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088310    2570 scope.go:117] "RemoveContainer" containerID="6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"
Sep 09 10:32:12.088574 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:32:12.088494    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1\": container with ID starting with 6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1 not found: ID does not exist" containerID="6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"
Sep 09 10:32:12.088574 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088510    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"} err="failed to get container status \"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1\": rpc error: code = NotFound desc = could not find container \"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1\": container with ID starting with 6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1 not found: ID does not exist"
Sep 09 10:32:12.088574 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088520    2570 scope.go:117] "RemoveContainer" containerID="3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"
Sep 09 10:32:12.088778 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:32:12.088746    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a\": container with ID starting with 3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a not found: ID does not exist" containerID="3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"
Sep 09 10:32:12.088824 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088804    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"} err="failed to get container status \"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a\": rpc error: code = NotFound desc = could not find container \"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a\": container with ID starting with 3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a not found: ID does not exist"
Sep 09 10:32:12.088824 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.088822    2570 scope.go:117] "RemoveContainer" containerID="6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"
Sep 09 10:32:12.089063 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:32:12.089045    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650\": container with ID starting with 6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650 not found: ID does not exist" containerID="6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"
Sep 09 10:32:12.089102 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.089069    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"} err="failed to get container status \"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650\": rpc error: code = NotFound desc = could not find container \"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650\": container with ID starting with 6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650 not found: ID does not exist"
Sep 09 10:32:12.089102 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.089085    2570 scope.go:117] "RemoveContainer" containerID="4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"
Sep 09 10:32:12.089353 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.089333    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"} err="failed to get container status \"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa\": rpc error: code = NotFound desc = could not find container \"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa\": container with ID starting with 4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa not found: ID does not exist"
Sep 09 10:32:12.089441 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.089354    2570 scope.go:117] "RemoveContainer" containerID="3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"
Sep 09 10:32:12.089624 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.089606    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"} err="failed to get container status \"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f\": rpc error: code = NotFound desc = could not find container \"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f\": container with ID starting with 3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f not found: ID does not exist"
Sep 09 10:32:12.089675 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.089624    2570 scope.go:117] "RemoveContainer" containerID="2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"
Sep 09 10:32:12.089869 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.089851    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"} err="failed to get container status \"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d\": rpc error: code = NotFound desc = could not find container \"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d\": container with ID starting with 2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d not found: ID does not exist"
Sep 09 10:32:12.089913 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.089870    2570 scope.go:117] "RemoveContainer" containerID="495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"
Sep 09 10:32:12.090116 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.090101    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"} err="failed to get container status \"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc\": rpc error: code = NotFound desc = could not find container \"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc\": container with ID starting with 495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc not found: ID does not exist"
Sep 09 10:32:12.090158 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.090115    2570 scope.go:117] "RemoveContainer" containerID="6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"
Sep 09 10:32:12.090305 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.090288    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"} err="failed to get container status \"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1\": rpc error: code = NotFound desc = could not find container \"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1\": container with ID starting with 6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1 not found: ID does not exist"
Sep 09 10:32:12.090344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.090307    2570 scope.go:117] "RemoveContainer" containerID="3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"
Sep 09 10:32:12.090512 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.090497    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"} err="failed to get container status \"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a\": rpc error: code = NotFound desc = could not find container \"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a\": container with ID starting with 3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a not found: ID does not exist"
Sep 09 10:32:12.090512 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.090510    2570 scope.go:117] "RemoveContainer" containerID="6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"
Sep 09 10:32:12.090685 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.090670    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"} err="failed to get container status \"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650\": rpc error: code = NotFound desc = could not find container \"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650\": container with ID starting with 6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650 not found: ID does not exist"
Sep 09 10:32:12.090685 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.090684    2570 scope.go:117] "RemoveContainer" containerID="4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"
Sep 09 10:32:12.090911 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.090894    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"} err="failed to get container status \"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa\": rpc error: code = NotFound desc = could not find container \"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa\": container with ID starting with 4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa not found: ID does not exist"
Sep 09 10:32:12.090967 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.090913    2570 scope.go:117] "RemoveContainer" containerID="3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"
Sep 09 10:32:12.091128 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.091114    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"} err="failed to get container status \"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f\": rpc error: code = NotFound desc = could not find container \"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f\": container with ID starting with 3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f not found: ID does not exist"
Sep 09 10:32:12.091172 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.091128    2570 scope.go:117] "RemoveContainer" containerID="2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"
Sep 09 10:32:12.091326 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.091312    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"} err="failed to get container status \"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d\": rpc error: code = NotFound desc = could not find container \"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d\": container with ID starting with 2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d not found: ID does not exist"
Sep 09 10:32:12.091369 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.091326    2570 scope.go:117] "RemoveContainer" containerID="495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"
Sep 09 10:32:12.091513 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.091497    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"} err="failed to get container status \"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc\": rpc error: code = NotFound desc = could not find container \"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc\": container with ID starting with 495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc not found: ID does not exist"
Sep 09 10:32:12.091553 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.091512    2570 scope.go:117] "RemoveContainer" containerID="6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"
Sep 09 10:32:12.091705 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.091690    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"} err="failed to get container status \"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1\": rpc error: code = NotFound desc = could not find container \"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1\": container with ID starting with 6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1 not found: ID does not exist"
Sep 09 10:32:12.091747 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.091705    2570 scope.go:117] "RemoveContainer" containerID="3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"
Sep 09 10:32:12.091936 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.091920    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"} err="failed to get container status \"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a\": rpc error: code = NotFound desc = could not find container \"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a\": container with ID starting with 3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a not found: ID does not exist"
Sep 09 10:32:12.091936 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.091935    2570 scope.go:117] "RemoveContainer" containerID="6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"
Sep 09 10:32:12.092122 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.092106    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"} err="failed to get container status \"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650\": rpc error: code = NotFound desc = could not find container \"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650\": container with ID starting with 6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650 not found: ID does not exist"
Sep 09 10:32:12.092162 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.092122    2570 scope.go:117] "RemoveContainer" containerID="4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"
Sep 09 10:32:12.092313 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.092298    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"} err="failed to get container status \"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa\": rpc error: code = NotFound desc = could not find container \"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa\": container with ID starting with 4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa not found: ID does not exist"
Sep 09 10:32:12.092355 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.092312    2570 scope.go:117] "RemoveContainer" containerID="3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"
Sep 09 10:32:12.092515 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.092500    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"} err="failed to get container status \"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f\": rpc error: code = NotFound desc = could not find container \"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f\": container with ID starting with 3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f not found: ID does not exist"
Sep 09 10:32:12.092515 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.092514    2570 scope.go:117] "RemoveContainer" containerID="2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"
Sep 09 10:32:12.092681 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.092666    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"} err="failed to get container status \"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d\": rpc error: code = NotFound desc = could not find container \"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d\": container with ID starting with 2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d not found: ID does not exist"
Sep 09 10:32:12.092681 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.092680    2570 scope.go:117] "RemoveContainer" containerID="495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"
Sep 09 10:32:12.092898 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.092882    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"} err="failed to get container status \"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc\": rpc error: code = NotFound desc = could not find container \"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc\": container with ID starting with 495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc not found: ID does not exist"
Sep 09 10:32:12.092951 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.092898    2570 scope.go:117] "RemoveContainer" containerID="6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"
Sep 09 10:32:12.093092 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.093077    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"} err="failed to get container status \"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1\": rpc error: code = NotFound desc = could not find container \"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1\": container with ID starting with 6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1 not found: ID does not exist"
Sep 09 10:32:12.093132 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.093094    2570 scope.go:117] "RemoveContainer" containerID="3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"
Sep 09 10:32:12.093335 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.093316    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"} err="failed to get container status \"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a\": rpc error: code = NotFound desc = could not find container \"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a\": container with ID starting with 3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a not found: ID does not exist"
Sep 09 10:32:12.093383 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.093335    2570 scope.go:117] "RemoveContainer" containerID="6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"
Sep 09 10:32:12.093550 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.093536    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"} err="failed to get container status \"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650\": rpc error: code = NotFound desc = could not find container \"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650\": container with ID starting with 6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650 not found: ID does not exist"
Sep 09 10:32:12.093550 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.093550    2570 scope.go:117] "RemoveContainer" containerID="4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"
Sep 09 10:32:12.093798 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.093780    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"} err="failed to get container status \"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa\": rpc error: code = NotFound desc = could not find container \"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa\": container with ID starting with 4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa not found: ID does not exist"
Sep 09 10:32:12.093798 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.093797    2570 scope.go:117] "RemoveContainer" containerID="3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"
Sep 09 10:32:12.094046 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.094030    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"} err="failed to get container status \"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f\": rpc error: code = NotFound desc = could not find container \"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f\": container with ID starting with 3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f not found: ID does not exist"
Sep 09 10:32:12.094094 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.094046    2570 scope.go:117] "RemoveContainer" containerID="2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"
Sep 09 10:32:12.094340 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.094318    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"} err="failed to get container status \"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d\": rpc error: code = NotFound desc = could not find container \"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d\": container with ID starting with 2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d not found: ID does not exist"
Sep 09 10:32:12.094390 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.094341    2570 scope.go:117] "RemoveContainer" containerID="495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"
Sep 09 10:32:12.094589 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.094568    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"} err="failed to get container status \"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc\": rpc error: code = NotFound desc = could not find container \"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc\": container with ID starting with 495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc not found: ID does not exist"
Sep 09 10:32:12.094589 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.094589    2570 scope.go:117] "RemoveContainer" containerID="6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"
Sep 09 10:32:12.094856 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.094835    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"} err="failed to get container status \"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1\": rpc error: code = NotFound desc = could not find container \"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1\": container with ID starting with 6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1 not found: ID does not exist"
Sep 09 10:32:12.094856 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.094856    2570 scope.go:117] "RemoveContainer" containerID="3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"
Sep 09 10:32:12.095132 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.095114    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"} err="failed to get container status \"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a\": rpc error: code = NotFound desc = could not find container \"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a\": container with ID starting with 3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a not found: ID does not exist"
Sep 09 10:32:12.095187 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.095133    2570 scope.go:117] "RemoveContainer" containerID="6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"
Sep 09 10:32:12.095436 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.095412    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"} err="failed to get container status \"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650\": rpc error: code = NotFound desc = could not find container \"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650\": container with ID starting with 6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650 not found: ID does not exist"
Sep 09 10:32:12.095511 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.095438    2570 scope.go:117] "RemoveContainer" containerID="4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"
Sep 09 10:32:12.095724 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.095702    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa"} err="failed to get container status \"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa\": rpc error: code = NotFound desc = could not find container \"4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa\": container with ID starting with 4e2c50a7ba24cc22495b7b08a6a21d889d67ec217413a425447647a5d5f9f2fa not found: ID does not exist"
Sep 09 10:32:12.095809 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.095726    2570 scope.go:117] "RemoveContainer" containerID="3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"
Sep 09 10:32:12.095865 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.095815    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 10:32:12.096002 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.095983    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f"} err="failed to get container status \"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f\": rpc error: code = NotFound desc = could not find container \"3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f\": container with ID starting with 3ab395a0e5b058d4c072d66a52bb8b27452cdbc11f3a7d5284e053d513cb504f not found: ID does not exist"
Sep 09 10:32:12.096046 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096002    2570 scope.go:117] "RemoveContainer" containerID="2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"
Sep 09 10:32:12.096086 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096061    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="kube-rbac-proxy-thanos"
Sep 09 10:32:12.096086 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096074    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="kube-rbac-proxy-thanos"
Sep 09 10:32:12.096165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096087    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf4831f-45ff-4feb-9aad-9237fae12648" containerName="console"
Sep 09 10:32:12.096165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096095    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf4831f-45ff-4feb-9aad-9237fae12648" containerName="console"
Sep 09 10:32:12.096165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096109    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="kube-rbac-proxy-web"
Sep 09 10:32:12.096165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096117    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="kube-rbac-proxy-web"
Sep 09 10:32:12.096165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096127    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="init-config-reloader"
Sep 09 10:32:12.096165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096135    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="init-config-reloader"
Sep 09 10:32:12.096165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096146    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="kube-rbac-proxy"
Sep 09 10:32:12.096165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096154    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="kube-rbac-proxy"
Sep 09 10:32:12.096165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096165    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="thanos-sidecar"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096172    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="thanos-sidecar"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096182    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="config-reloader"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096190    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="config-reloader"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096200    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe4c67ed-5d54-4d1c-84b4-febd3326c935" containerName="registry"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096208    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe4c67ed-5d54-4d1c-84b4-febd3326c935" containerName="registry"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096216    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="prometheus"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096220    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="prometheus"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096270    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="config-reloader"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096277    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="thanos-sidecar"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096269    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d"} err="failed to get container status \"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d\": rpc error: code = NotFound desc = could not find container \"2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d\": container with ID starting with 2ffd9856a2918947874d6099f1950f61b5f875fb71897a2683997bec085c8e4d not found: ID does not exist"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096299    2570 scope.go:117] "RemoveContainer" containerID="495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096282    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="kube-rbac-proxy-web"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096370    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="prometheus"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096381    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe4c67ed-5d54-4d1c-84b4-febd3326c935" containerName="registry"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096392    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bf4831f-45ff-4feb-9aad-9237fae12648" containerName="console"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096402    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="kube-rbac-proxy"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096413    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" containerName="kube-rbac-proxy-thanos"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096522    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc"} err="failed to get container status \"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc\": rpc error: code = NotFound desc = could not find container \"495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc\": container with ID starting with 495d29e95e4a60dbc7cd7b6bf375ccbc537b57838cc52c6d42c48695a92050cc not found: ID does not exist"
Sep 09 10:32:12.096534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096535    2570 scope.go:117] "RemoveContainer" containerID="6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"
Sep 09 10:32:12.097445 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096758    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1"} err="failed to get container status \"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1\": rpc error: code = NotFound desc = could not find container \"6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1\": container with ID starting with 6ad8d94e5f4369a9247abed0bfaa6da9ef128ed5d182210154576e2762f532a1 not found: ID does not exist"
Sep 09 10:32:12.097445 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.096824    2570 scope.go:117] "RemoveContainer" containerID="3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"
Sep 09 10:32:12.097445 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.097026    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a"} err="failed to get container status \"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a\": rpc error: code = NotFound desc = could not find container \"3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a\": container with ID starting with 3f219b636cacdabbea613f38bad832d49aebbc4a413d9f1ccfed5a9c351e0c5a not found: ID does not exist"
Sep 09 10:32:12.097445 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.097044    2570 scope.go:117] "RemoveContainer" containerID="6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"
Sep 09 10:32:12.097445 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.097243    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650"} err="failed to get container status \"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650\": rpc error: code = NotFound desc = could not find container \"6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650\": container with ID starting with 6ce93b143642697e1727869888b7f5b5a1e7cbfef282f885af63813f8e930650 not found: ID does not exist"
Sep 09 10:32:12.100677 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.100660    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.103453 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.103436    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 10:32:12.103688 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.103668    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 10:32:12.103798 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.103716    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 10:32:12.103798 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.103737    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 10:32:12.103920 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.103901    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aqdtqku09ggaj\""
Sep 09 10:32:12.104116 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.104102    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 10:32:12.104116 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.104110    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 10:32:12.104225 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.104156    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 10:32:12.104225 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.104174    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 10:32:12.104319 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.104299    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 10:32:12.104419 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.104403    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 10:32:12.104549 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.104497    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 10:32:12.104549 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.104514    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-m4r6v\""
Sep 09 10:32:12.104549 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.104525    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 10:32:12.104844 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.104528    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 10:32:12.108164 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.108147    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 10:32:12.109711 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.109695    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 10:32:12.121322 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.121298    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 10:32:12.188928 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.188893    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.188928 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.188927    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189100 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.188947    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x44x\" (UniqueName: \"kubernetes.io/projected/0a32f131-bb91-4a70-a75c-5873fcc41367-kube-api-access-8x44x\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189100 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.188983    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189100 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189018    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189100 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189036    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-config\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189100 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189049    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-web-config\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189099    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189138    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189170    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189191    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a32f131-bb91-4a70-a75c-5873fcc41367-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189218    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189241    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189429 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189289    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189429 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189307    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189429 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189321    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a32f131-bb91-4a70-a75c-5873fcc41367-config-out\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189429 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189337    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189428    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189456    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.189545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.189471    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290270 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290231    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a32f131-bb91-4a70-a75c-5873fcc41367-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290270 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290270    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290468 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290288    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290468 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290311    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290468 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290335    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290468 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290360    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a32f131-bb91-4a70-a75c-5873fcc41367-config-out\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290468 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290382    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290468 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290426    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290614    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290650    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290690    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290725    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290987 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290761    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x44x\" (UniqueName: \"kubernetes.io/projected/0a32f131-bb91-4a70-a75c-5873fcc41367-kube-api-access-8x44x\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290987 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290821    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290987 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290857    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290987 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290885    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-config\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290987 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290906    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-web-config\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290987 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290934    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.290987 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290961    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.291309 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.290995    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.291309 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.291109    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.294344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.293338    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a32f131-bb91-4a70-a75c-5873fcc41367-config-out\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.294344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.293356    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a32f131-bb91-4a70-a75c-5873fcc41367-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.294344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.293558    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.294344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.293578    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.294344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.293689    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.294344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.293792    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.294344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.293895    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.294344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.294032    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.294344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.294134    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.294344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.294160    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.294344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.294293    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.294344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.294331    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.295026 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.294382    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.295026 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.294794    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.295726 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.295704    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-config\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.295836 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.295795    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.296042 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.296024    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a32f131-bb91-4a70-a75c-5873fcc41367-web-config\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.296783 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.296737    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0a32f131-bb91-4a70-a75c-5873fcc41367-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.300579 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.300563    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x44x\" (UniqueName: \"kubernetes.io/projected/0a32f131-bb91-4a70-a75c-5873fcc41367-kube-api-access-8x44x\") pod \"prometheus-k8s-0\" (UID: \"0a32f131-bb91-4a70-a75c-5873fcc41367\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.410428 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.410395    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:12.686427 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.686345    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 10:32:12.688674 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:32:12.688644    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a32f131_bb91_4a70_a75c_5873fcc41367.slice/crio-2d1e48c1b79d429e7c5b7188af89a5bc0991a083fbcf6069950c2082111371bf WatchSource:0}: Error finding container 2d1e48c1b79d429e7c5b7188af89a5bc0991a083fbcf6069950c2082111371bf: Status 404 returned error can't find the container with id 2d1e48c1b79d429e7c5b7188af89a5bc0991a083fbcf6069950c2082111371bf
Sep 09 10:32:12.838401 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:12.838365    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3afa275a-2875-431c-bb30-10ef6d088c97" path="/var/lib/kubelet/pods/3afa275a-2875-431c-bb30-10ef6d088c97/volumes"
Sep 09 10:32:13.035181 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:13.035149    2570 generic.go:356] "Generic (PLEG): container finished" podID="0a32f131-bb91-4a70-a75c-5873fcc41367" containerID="748b50043754ef600fe919b75d7f606fe7d4643553a6bde4949e8463232f136b" exitCode=0
Sep 09 10:32:13.035336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:13.035188    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a32f131-bb91-4a70-a75c-5873fcc41367","Type":"ContainerDied","Data":"748b50043754ef600fe919b75d7f606fe7d4643553a6bde4949e8463232f136b"}
Sep 09 10:32:13.035791 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:13.035631    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a32f131-bb91-4a70-a75c-5873fcc41367","Type":"ContainerStarted","Data":"2d1e48c1b79d429e7c5b7188af89a5bc0991a083fbcf6069950c2082111371bf"}
Sep 09 10:32:14.048368 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:14.048335    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a32f131-bb91-4a70-a75c-5873fcc41367","Type":"ContainerStarted","Data":"76b324fcc47e2a45f78b1846ce6a770a2e3030cfd4ac0cc60ec6940edf413ef6"}
Sep 09 10:32:14.048733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:14.048397    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a32f131-bb91-4a70-a75c-5873fcc41367","Type":"ContainerStarted","Data":"9d05533200452ddebe46ab6b9c5ee16b143bf3c8fa1df3b0bcc445a0c8322d05"}
Sep 09 10:32:14.048733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:14.048422    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a32f131-bb91-4a70-a75c-5873fcc41367","Type":"ContainerStarted","Data":"7fe12445c6a11619ba679861a79bfd0ace4fd9e9f48b46044c27bcfc6d2773ac"}
Sep 09 10:32:14.048733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:14.048447    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a32f131-bb91-4a70-a75c-5873fcc41367","Type":"ContainerStarted","Data":"57ad1e16918b9f84b310abff4a55f841e9b2d520af4de862d76f8fe0fcb9b136"}
Sep 09 10:32:14.048733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:14.048469    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a32f131-bb91-4a70-a75c-5873fcc41367","Type":"ContainerStarted","Data":"70c1dbf95e36d0ba5394f6f5ef131c39be6218185eba3c2ec68b5f05cb84cf21"}
Sep 09 10:32:14.048733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:14.048492    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a32f131-bb91-4a70-a75c-5873fcc41367","Type":"ContainerStarted","Data":"7e2a215f30154c438256c857cb36f39ee94b0e21f98b3eeeaa97547d583ca0b7"}
Sep 09 10:32:14.078714 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:14.078667    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.061015411 podStartE2EDuration="2.078650079s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.634668ms" isStatefulPod=false podCreationTimestamp="2026-09-09 10:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:32:14.076681955 +0000 UTC m=+373.740320941" watchObservedRunningTime="2026-09-09 10:32:14.078650079 +0000 UTC m=+373.742289065"
Sep 09 10:32:14.714525 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:14.714484    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:32:14.716798 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:14.716752    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969769c-d30e-4cbb-afc5-9df38c741545-metrics-certs\") pod \"network-metrics-daemon-llgq5\" (UID: \"6969769c-d30e-4cbb-afc5-9df38c741545\") " pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:32:14.937327 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:14.937296    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:14.937653 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:14.937640    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hq5pc\""
Sep 09 10:32:14.945823 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:14.945804    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-llgq5"
Sep 09 10:32:15.003616 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:15.003582    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:15.217754 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:15.217691    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-llgq5"]
Sep 09 10:32:15.220313 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:32:15.220280    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6969769c_d30e_4cbb_afc5_9df38c741545.slice/crio-1fe7ebb5e1834142f601e70d556cba405f63392860728ecb2bd0c0c0ddc58980 WatchSource:0}: Error finding container 1fe7ebb5e1834142f601e70d556cba405f63392860728ecb2bd0c0c0ddc58980: Status 404 returned error can't find the container with id 1fe7ebb5e1834142f601e70d556cba405f63392860728ecb2bd0c0c0ddc58980
Sep 09 10:32:15.222492 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:15.222476    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 10:32:15.810503 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:15.810464    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:15.925460 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:15.925381    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:16.001974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:16.001940    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:16.058024 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:16.057979    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-llgq5" event={"ID":"6969769c-d30e-4cbb-afc5-9df38c741545","Type":"ContainerStarted","Data":"1fe7ebb5e1834142f601e70d556cba405f63392860728ecb2bd0c0c0ddc58980"}
Sep 09 10:32:16.147913 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:16.147826    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:16.217233 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:16.217194    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:17.065879 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:17.065844    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-llgq5" event={"ID":"6969769c-d30e-4cbb-afc5-9df38c741545","Type":"ContainerStarted","Data":"4330b9d0f53b350c9394438e182fa06f7f73d6d4b0bb497909f0c0d0536135c8"}
Sep 09 10:32:17.066347 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:17.065902    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-llgq5" event={"ID":"6969769c-d30e-4cbb-afc5-9df38c741545","Type":"ContainerStarted","Data":"b64fede6a108503025ff157e7ba8fba1f4cd1ff0eb80097dbcb6b4db50a41275"}
Sep 09 10:32:17.084931 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:17.084886    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-llgq5" podStartSLOduration=374.063793905 podStartE2EDuration="6m15.084870305s" totalImagesPullingTime="1.0210764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:32:17.082827512 +0000 UTC m=+376.746466490" watchObservedRunningTime="2026-09-09 10:32:17.084870305 +0000 UTC m=+376.748509278"
Sep 09 10:32:17.188611 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:17.188567    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:17.410914 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:17.410831    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:32:17.540195 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:17.540161    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:17.640211 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:17.640172    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:25.141505 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:25.141451    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:25.983405 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:25.983373    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:26.083660 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:26.083605    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:26.149181 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:26.149137    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:52.199693 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:52.199635    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:32:52.311618 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:32:52.311589    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:04.076143 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:04.076088    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:12.410978 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:12.410925    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:33:12.425989 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:12.425966    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:33:13.284382 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:13.284356    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 10:33:21.579252 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:21.579211    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:21.647167 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:21.647102    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:28.225470 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:28.225362    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:28.321383 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:28.321343    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:28.514041 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:28.514010    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:28.932176 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:28.932087    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:28.998317 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:28.998283    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:33.520915 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:33.520882    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:33.624197 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:33.624145    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:39.286005 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:39.285974    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:39.386653 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:39.386622    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:39.487489 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:39.487461    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:42.902699 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:42.902662    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:33:43.005137 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:33:43.005100    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:34:21.976457 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:34:21.976426    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:34:22.366394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:34:22.366362    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:35:20.179003 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:35:20.178961    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:35:54.008612 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:35:54.008571    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:35:54.074619 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:35:54.074592    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:35:54.154512 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:35:54.154482    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:35:54.220698 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:35:54.220669    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:35:54.745044 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:35:54.745012    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:35:54.834652 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:35:54.834621    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:00.992168 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:00.992138    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:36:00.993445 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:00.993423    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:36:00.998499 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:00.998473    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:36:01.000073 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:01.000055    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:36:04.911742 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:04.911709    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:07.093585 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:07.093550    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:08.462155 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:08.462126    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:09.814258 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:09.814230    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:12.276012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:12.275979    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:12.380096 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:12.380060    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:13.720991 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:13.720963    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:13.815396 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:13.815360    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:14.185915 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:14.185887    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:14.341321 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:14.341292    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:15.946980 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:15.946946    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:16.338501 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:16.338469    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:19.072473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:19.072432    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:19.151135 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:19.151078    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:23.958346 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:23.958272    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:24.059551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:24.059524    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:24.527917 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:24.527881    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:26.749782 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:26.749737    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:26.844822 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:26.844797    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:32.568968 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:32.568941    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:32.668795 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:32.668750    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:34.526475 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:34.526445    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:50.510592 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:50.510557    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:50.600118 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:50.600074    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:51.184857 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.184824    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:51.438534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.438458    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:51.560690 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.560659    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:51.575398 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.575369    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"]
Sep 09 10:36:51.578581 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.578567    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.581521 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.581503    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 10:36:51.581859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.581841    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 10:36:51.581949 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.581878    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 10:36:51.582133 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.582117    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 10:36:51.582493 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.582477    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-llqpt\""
Sep 09 10:36:51.596159 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.596135    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"]
Sep 09 10:36:51.652578 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.652549    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cde22cf6-342a-438c-948a-be71c8ad26c3-tmp\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.652704 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.652582    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cde22cf6-342a-438c-948a-be71c8ad26c3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.652704 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.652653    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cde22cf6-342a-438c-948a-be71c8ad26c3-webhook-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.652812 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.652717    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cde22cf6-342a-438c-948a-be71c8ad26c3-manifests\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.652812 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.652741    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxfzb\" (UniqueName: \"kubernetes.io/projected/cde22cf6-342a-438c-948a-be71c8ad26c3-kube-api-access-sxfzb\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.733504 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.733421    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:51.753506 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.753485    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cde22cf6-342a-438c-948a-be71c8ad26c3-manifests\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.753568 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.753516    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxfzb\" (UniqueName: \"kubernetes.io/projected/cde22cf6-342a-438c-948a-be71c8ad26c3-kube-api-access-sxfzb\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.753568 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.753550    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cde22cf6-342a-438c-948a-be71c8ad26c3-tmp\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.753670 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.753571    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cde22cf6-342a-438c-948a-be71c8ad26c3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.753670 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.753598    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cde22cf6-342a-438c-948a-be71c8ad26c3-webhook-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.753943 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.753922    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cde22cf6-342a-438c-948a-be71c8ad26c3-manifests\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.755900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.755870    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cde22cf6-342a-438c-948a-be71c8ad26c3-tmp\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.755986 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.755919    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cde22cf6-342a-438c-948a-be71c8ad26c3-webhook-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.756029 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.756008    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cde22cf6-342a-438c-948a-be71c8ad26c3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.763322 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.763302    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxfzb\" (UniqueName: \"kubernetes.io/projected/cde22cf6-342a-438c-948a-be71c8ad26c3-kube-api-access-sxfzb\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2lx89\" (UID: \"cde22cf6-342a-438c-948a-be71c8ad26c3\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.836841 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.836813    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:51.889097 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.889076    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:51.954555 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:51.954517    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:52.051690 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:52.051646    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:52.152210 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:52.152163    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:52.175421 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:52.175401    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"]
Sep 09 10:36:52.176472 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:36:52.176450    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde22cf6_342a_438c_948a_be71c8ad26c3.slice/crio-1af0267e0a6097a573a6e4edac3072fe2ba0598b98440c08e12405e27140dd56 WatchSource:0}: Error finding container 1af0267e0a6097a573a6e4edac3072fe2ba0598b98440c08e12405e27140dd56: Status 404 returned error can't find the container with id 1af0267e0a6097a573a6e4edac3072fe2ba0598b98440c08e12405e27140dd56
Sep 09 10:36:53.048208 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:53.048173    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89" event={"ID":"cde22cf6-342a-438c-948a-be71c8ad26c3","Type":"ContainerStarted","Data":"1af0267e0a6097a573a6e4edac3072fe2ba0598b98440c08e12405e27140dd56"}
Sep 09 10:36:56.061180 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:56.061136    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89" event={"ID":"cde22cf6-342a-438c-948a-be71c8ad26c3","Type":"ContainerStarted","Data":"6e05da7f060730eae3835048c37924fad3e3d6354901263ea39bf449f95a586f"}
Sep 09 10:36:56.835915 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:56.835882    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:56.935591 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:56.935562    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:57.066126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:57.066099    2570 generic.go:356] "Generic (PLEG): container finished" podID="cde22cf6-342a-438c-948a-be71c8ad26c3" containerID="6e05da7f060730eae3835048c37924fad3e3d6354901263ea39bf449f95a586f" exitCode=0
Sep 09 10:36:57.066478 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:57.066145    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89" event={"ID":"cde22cf6-342a-438c-948a-be71c8ad26c3","Type":"ContainerDied","Data":"6e05da7f060730eae3835048c37924fad3e3d6354901263ea39bf449f95a586f"}
Sep 09 10:36:58.071190 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:58.071155    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89" event={"ID":"cde22cf6-342a-438c-948a-be71c8ad26c3","Type":"ContainerStarted","Data":"93a1125040df82ca071032b0ae9f25dc9269724d1501734e0e7b84bb53522c89"}
Sep 09 10:36:58.071732 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:58.071266    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:36:58.095311 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:58.095259    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89" podStartSLOduration=2.7288915 podStartE2EDuration="7.095245248s" totalImagesPullingTime="3.333386007s" totalInitContainerRuntime="1.032967741s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:36:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:36:58.093867734 +0000 UTC m=+657.757506709" watchObservedRunningTime="2026-09-09 10:36:58.095245248 +0000 UTC m=+657.758884222"
Sep 09 10:36:58.213200 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:58.213158    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:36:58.310078 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:36:58.310047    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:03.091364 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:03.091288    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:03.187025 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:03.186984    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:09.077085 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:09.077053    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2lx89"
Sep 09 10:37:11.664016 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:11.663955    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:11.784890 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:11.784859    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:12.092832 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:12.092801    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:12.323499 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:12.323466    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:14.042703 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.042672    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:14.053756 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.053726    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"]
Sep 09 10:37:14.064543 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.064524    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.067185 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.067165    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 10:37:14.067277 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.067184    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 10:37:14.067277 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.067214    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-r4662\""
Sep 09 10:37:14.067440 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.067427    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 10:37:14.067515 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.067433    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 10:37:14.073146 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.073122    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"]
Sep 09 10:37:14.136713 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.136688    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:14.240074 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.240034    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.240074 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.240076    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.240240 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.240092    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbw6\" (UniqueName: \"kubernetes.io/projected/b9a14fb7-7e57-4377-8000-a6075a5a3f14-kube-api-access-xnbw6\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.240240 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.240182    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.240240 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.240211    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.240240 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.240234    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a14fb7-7e57-4377-8000-a6075a5a3f14-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.240368 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.240280    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.341521 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.341424    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.341521 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.341475    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.341521 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.341498    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a14fb7-7e57-4377-8000-a6075a5a3f14-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.341521 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.341524    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.341864 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.341575    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.341864 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.341593    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.341864 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.341609    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbw6\" (UniqueName: \"kubernetes.io/projected/b9a14fb7-7e57-4377-8000-a6075a5a3f14-kube-api-access-xnbw6\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.342229 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.342199    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.343832 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.343807    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a14fb7-7e57-4377-8000-a6075a5a3f14-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.343948 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.343932    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.344292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.344274    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.344329 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.344274    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.351650 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.351626    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.351925 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.351907    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbw6\" (UniqueName: \"kubernetes.io/projected/b9a14fb7-7e57-4377-8000-a6075a5a3f14-kube-api-access-xnbw6\") pod \"istiod-openshift-gateway-7b7995d5cd-dgnsg\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.374688 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.374667    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:14.646835 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:14.646812    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"]
Sep 09 10:37:14.648891 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:37:14.648867    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a14fb7_7e57_4377_8000_a6075a5a3f14.slice/crio-e57e74654ed718ea92522e60d0c0366d10b91195f8c5f10f2ce57f3be3c5c1b7 WatchSource:0}: Error finding container e57e74654ed718ea92522e60d0c0366d10b91195f8c5f10f2ce57f3be3c5c1b7: Status 404 returned error can't find the container with id e57e74654ed718ea92522e60d0c0366d10b91195f8c5f10f2ce57f3be3c5c1b7
Sep 09 10:37:15.131680 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:15.131647    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg" event={"ID":"b9a14fb7-7e57-4377-8000-a6075a5a3f14","Type":"ContainerStarted","Data":"e57e74654ed718ea92522e60d0c0366d10b91195f8c5f10f2ce57f3be3c5c1b7"}
Sep 09 10:37:15.204750 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:15.204717    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:17.278122 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:17.278091    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:37:17.278377 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:17.278152    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:37:17.905147 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:17.905115    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:18.005818 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:18.005759    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:18.144903 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:18.144871    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg" event={"ID":"b9a14fb7-7e57-4377-8000-a6075a5a3f14","Type":"ContainerStarted","Data":"37689595450bab6e055f8cdcf5842f2cff9695bbad1c52c8e440bb94ad459e78"}
Sep 09 10:37:18.145134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:18.145111    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:18.147107 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:18.147083    2570 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-dgnsg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 10:37:18.147184 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:18.147130    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg" podUID="b9a14fb7-7e57-4377-8000-a6075a5a3f14" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 10:37:18.174208 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:18.174120    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg" podStartSLOduration=1.5470633440000001 podStartE2EDuration="4.174101221s" totalImagesPullingTime="2.627037877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:37:18.17308661 +0000 UTC m=+677.836725596" watchObservedRunningTime="2026-09-09 10:37:18.174101221 +0000 UTC m=+677.837740196"
Sep 09 10:37:18.223783 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:18.223730    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:18.378631 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:18.378598    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:18.701019 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:18.700988    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:19.148846 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:19.148820    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:22.655535 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:22.655510    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:22.746116 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:22.746089    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:22.848948 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:22.848917    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:22.947435 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:22.947347    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:23.232321 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:23.232238    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:23.462087 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:23.462052    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:23.781910 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:23.781876    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:23.882589 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:23.882557    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:30.627838 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.627796    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"]
Sep 09 10:37:30.630931 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.630913    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:30.633759 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.633730    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 10:37:30.634855 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.634834    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 10:37:30.634980 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.634957    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 10:37:30.635054 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.634984    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ch6ct\""
Sep 09 10:37:30.635123 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.635063    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 10:37:30.635123 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.635062    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 10:37:30.644700 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.644680    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"]
Sep 09 10:37:30.659415 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.659389    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c93c33a-2f17-4516-b59d-d639733c89bd-cert\") pod \"lws-controller-manager-7658745f98-zpbnk\" (UID: \"5c93c33a-2f17-4516-b59d-d639733c89bd\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:30.659520 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.659437    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5c93c33a-2f17-4516-b59d-d639733c89bd-manager-config\") pod \"lws-controller-manager-7658745f98-zpbnk\" (UID: \"5c93c33a-2f17-4516-b59d-d639733c89bd\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:30.659586 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.659565    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5bxj\" (UniqueName: \"kubernetes.io/projected/5c93c33a-2f17-4516-b59d-d639733c89bd-kube-api-access-z5bxj\") pod \"lws-controller-manager-7658745f98-zpbnk\" (UID: \"5c93c33a-2f17-4516-b59d-d639733c89bd\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:30.659639 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.659603    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c93c33a-2f17-4516-b59d-d639733c89bd-metrics-cert\") pod \"lws-controller-manager-7658745f98-zpbnk\" (UID: \"5c93c33a-2f17-4516-b59d-d639733c89bd\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:30.760308 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.760273    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5c93c33a-2f17-4516-b59d-d639733c89bd-manager-config\") pod \"lws-controller-manager-7658745f98-zpbnk\" (UID: \"5c93c33a-2f17-4516-b59d-d639733c89bd\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:30.760474 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.760340    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5bxj\" (UniqueName: \"kubernetes.io/projected/5c93c33a-2f17-4516-b59d-d639733c89bd-kube-api-access-z5bxj\") pod \"lws-controller-manager-7658745f98-zpbnk\" (UID: \"5c93c33a-2f17-4516-b59d-d639733c89bd\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:30.760474 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.760365    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c93c33a-2f17-4516-b59d-d639733c89bd-metrics-cert\") pod \"lws-controller-manager-7658745f98-zpbnk\" (UID: \"5c93c33a-2f17-4516-b59d-d639733c89bd\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:30.760474 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.760394    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c93c33a-2f17-4516-b59d-d639733c89bd-cert\") pod \"lws-controller-manager-7658745f98-zpbnk\" (UID: \"5c93c33a-2f17-4516-b59d-d639733c89bd\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:30.760968 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.760943    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5c93c33a-2f17-4516-b59d-d639733c89bd-manager-config\") pod \"lws-controller-manager-7658745f98-zpbnk\" (UID: \"5c93c33a-2f17-4516-b59d-d639733c89bd\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:30.762810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.762783    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c93c33a-2f17-4516-b59d-d639733c89bd-cert\") pod \"lws-controller-manager-7658745f98-zpbnk\" (UID: \"5c93c33a-2f17-4516-b59d-d639733c89bd\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:30.762988 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.762967    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c93c33a-2f17-4516-b59d-d639733c89bd-metrics-cert\") pod \"lws-controller-manager-7658745f98-zpbnk\" (UID: \"5c93c33a-2f17-4516-b59d-d639733c89bd\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:30.769115 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.769096    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5bxj\" (UniqueName: \"kubernetes.io/projected/5c93c33a-2f17-4516-b59d-d639733c89bd-kube-api-access-z5bxj\") pod \"lws-controller-manager-7658745f98-zpbnk\" (UID: \"5c93c33a-2f17-4516-b59d-d639733c89bd\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:30.940101 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:30.940023    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:31.208657 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:31.208636    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"]
Sep 09 10:37:31.211007 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:37:31.210981    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c93c33a_2f17_4516_b59d_d639733c89bd.slice/crio-411821710b54e939a5f2414c9d02b87d56ab2803c61eb54ffd41280ddc8e7d5b WatchSource:0}: Error finding container 411821710b54e939a5f2414c9d02b87d56ab2803c61eb54ffd41280ddc8e7d5b: Status 404 returned error can't find the container with id 411821710b54e939a5f2414c9d02b87d56ab2803c61eb54ffd41280ddc8e7d5b
Sep 09 10:37:31.213282 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:31.213265    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 10:37:32.016324 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:32.016248    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:32.128808 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:32.128756    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:32.199228 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:32.199189    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk" event={"ID":"5c93c33a-2f17-4516-b59d-d639733c89bd","Type":"ContainerStarted","Data":"411821710b54e939a5f2414c9d02b87d56ab2803c61eb54ffd41280ddc8e7d5b"}
Sep 09 10:37:32.692704 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:32.692661    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:32.819028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:32.818997    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:33.334052 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:33.334016    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:33.431953 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:33.431895    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:34.208232 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:34.208202    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk" event={"ID":"5c93c33a-2f17-4516-b59d-d639733c89bd","Type":"ContainerStarted","Data":"780f448e10f93837f1c947bc4379fbee8b1b5aa8e79f90c20332c1daeba9b048"}
Sep 09 10:37:34.208455 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:34.208312    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:37:34.234042 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:34.233997    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk" podStartSLOduration=1.928420558 podStartE2EDuration="4.233982362s" totalImagesPullingTime="2.305561804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:37:34.2331964 +0000 UTC m=+693.896835374" watchObservedRunningTime="2026-09-09 10:37:34.233982362 +0000 UTC m=+693.897621335"
Sep 09 10:37:34.684552 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:34.684516    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:34.784427 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:34.784398    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:34.883469 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:34.883439    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:37:39.919500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:39.919463    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"]
Sep 09 10:37:39.920071 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:39.919673    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg" podUID="b9a14fb7-7e57-4377-8000-a6075a5a3f14" containerName="discovery" containerID="cri-o://37689595450bab6e055f8cdcf5842f2cff9695bbad1c52c8e440bb94ad459e78" gracePeriod=30
Sep 09 10:37:40.159168 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.159143    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:40.236885 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.236344    2570 generic.go:356] "Generic (PLEG): container finished" podID="b9a14fb7-7e57-4377-8000-a6075a5a3f14" containerID="37689595450bab6e055f8cdcf5842f2cff9695bbad1c52c8e440bb94ad459e78" exitCode=0
Sep 09 10:37:40.236885 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.236620    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-cacerts\") pod \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") "
Sep 09 10:37:40.236885 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.236702    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-csr-ca-configmap\") pod \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") "
Sep 09 10:37:40.236885 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.236736    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a14fb7-7e57-4377-8000-a6075a5a3f14-local-certs\") pod \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") "
Sep 09 10:37:40.237908 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.237228    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-csr-dns-cert\") pod \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") "
Sep 09 10:37:40.237908 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.237266    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnbw6\" (UniqueName: \"kubernetes.io/projected/b9a14fb7-7e57-4377-8000-a6075a5a3f14-kube-api-access-xnbw6\") pod \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") "
Sep 09 10:37:40.237908 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.237305    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-token\") pod \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") "
Sep 09 10:37:40.237908 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.237327    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-kubeconfig\") pod \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\" (UID: \"b9a14fb7-7e57-4377-8000-a6075a5a3f14\") "
Sep 09 10:37:40.239268 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.238483    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b9a14fb7-7e57-4377-8000-a6075a5a3f14" (UID: "b9a14fb7-7e57-4377-8000-a6075a5a3f14"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:37:40.239268 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.238563    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"
Sep 09 10:37:40.239268 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.238859    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg" event={"ID":"b9a14fb7-7e57-4377-8000-a6075a5a3f14","Type":"ContainerDied","Data":"37689595450bab6e055f8cdcf5842f2cff9695bbad1c52c8e440bb94ad459e78"}
Sep 09 10:37:40.239268 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.238903    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg" event={"ID":"b9a14fb7-7e57-4377-8000-a6075a5a3f14","Type":"ContainerDied","Data":"e57e74654ed718ea92522e60d0c0366d10b91195f8c5f10f2ce57f3be3c5c1b7"}
Sep 09 10:37:40.239268 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.238925    2570 scope.go:117] "RemoveContainer" containerID="37689595450bab6e055f8cdcf5842f2cff9695bbad1c52c8e440bb94ad459e78"
Sep 09 10:37:40.245271 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.245223    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-token" (OuterVolumeSpecName: "istio-token") pod "b9a14fb7-7e57-4377-8000-a6075a5a3f14" (UID: "b9a14fb7-7e57-4377-8000-a6075a5a3f14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:37:40.247504 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.247465    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b9a14fb7-7e57-4377-8000-a6075a5a3f14" (UID: "b9a14fb7-7e57-4377-8000-a6075a5a3f14"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:37:40.249134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.249103    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a14fb7-7e57-4377-8000-a6075a5a3f14-kube-api-access-xnbw6" (OuterVolumeSpecName: "kube-api-access-xnbw6") pod "b9a14fb7-7e57-4377-8000-a6075a5a3f14" (UID: "b9a14fb7-7e57-4377-8000-a6075a5a3f14"). InnerVolumeSpecName "kube-api-access-xnbw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:37:40.260342 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.260146    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a14fb7-7e57-4377-8000-a6075a5a3f14-local-certs" (OuterVolumeSpecName: "local-certs") pod "b9a14fb7-7e57-4377-8000-a6075a5a3f14" (UID: "b9a14fb7-7e57-4377-8000-a6075a5a3f14"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:37:40.260342 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.260267    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-cacerts" (OuterVolumeSpecName: "cacerts") pod "b9a14fb7-7e57-4377-8000-a6075a5a3f14" (UID: "b9a14fb7-7e57-4377-8000-a6075a5a3f14"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:37:40.260991 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.260960    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b9a14fb7-7e57-4377-8000-a6075a5a3f14" (UID: "b9a14fb7-7e57-4377-8000-a6075a5a3f14"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:37:40.268713 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.268688    2570 scope.go:117] "RemoveContainer" containerID="37689595450bab6e055f8cdcf5842f2cff9695bbad1c52c8e440bb94ad459e78"
Sep 09 10:37:40.269128 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:37:40.269097    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37689595450bab6e055f8cdcf5842f2cff9695bbad1c52c8e440bb94ad459e78\": container with ID starting with 37689595450bab6e055f8cdcf5842f2cff9695bbad1c52c8e440bb94ad459e78 not found: ID does not exist" containerID="37689595450bab6e055f8cdcf5842f2cff9695bbad1c52c8e440bb94ad459e78"
Sep 09 10:37:40.269212 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.269135    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37689595450bab6e055f8cdcf5842f2cff9695bbad1c52c8e440bb94ad459e78"} err="failed to get container status \"37689595450bab6e055f8cdcf5842f2cff9695bbad1c52c8e440bb94ad459e78\": rpc error: code = NotFound desc = could not find container \"37689595450bab6e055f8cdcf5842f2cff9695bbad1c52c8e440bb94ad459e78\": container with ID starting with 37689595450bab6e055f8cdcf5842f2cff9695bbad1c52c8e440bb94ad459e78 not found: ID does not exist"
Sep 09 10:37:40.338326 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.338298    2570 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-cacerts\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:37:40.338326 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.338320    2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-csr-ca-configmap\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:37:40.338326 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.338330    2570 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a14fb7-7e57-4377-8000-a6075a5a3f14-local-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:37:40.338514 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.338339    2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-csr-dns-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:37:40.338514 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.338348    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnbw6\" (UniqueName: \"kubernetes.io/projected/b9a14fb7-7e57-4377-8000-a6075a5a3f14-kube-api-access-xnbw6\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:37:40.338514 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.338356    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:37:40.338514 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.338363    2570 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9a14fb7-7e57-4377-8000-a6075a5a3f14-istio-kubeconfig\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:37:40.562005 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.561977    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"]
Sep 09 10:37:40.566182 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.566158    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg"]
Sep 09 10:37:40.839087 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:40.839015    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9a14fb7-7e57-4377-8000-a6075a5a3f14" path="/var/lib/kubelet/pods/b9a14fb7-7e57-4377-8000-a6075a5a3f14/volumes"
Sep 09 10:37:45.149427 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:45.149392    2570 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-dgnsg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.20:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 10:37:45.149821 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:45.149459    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dgnsg" podUID="b9a14fb7-7e57-4377-8000-a6075a5a3f14" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.20:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 10:37:45.215026 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:37:45.215001    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7658745f98-zpbnk"
Sep 09 10:38:23.809123 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:23.809086    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:23.914235 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:23.914206    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:47.599818 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:47.599787    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:47.697328 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:47.697289    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:49.603796 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:49.603751    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-f54rk"]
Sep 09 10:38:49.604146 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:49.604032    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a14fb7-7e57-4377-8000-a6075a5a3f14" containerName="discovery"
Sep 09 10:38:49.604146 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:49.604043    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a14fb7-7e57-4377-8000-a6075a5a3f14" containerName="discovery"
Sep 09 10:38:49.604146 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:49.604089    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a14fb7-7e57-4377-8000-a6075a5a3f14" containerName="discovery"
Sep 09 10:38:49.607045 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:49.607030    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-f54rk"
Sep 09 10:38:49.609958 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:49.609934    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 10:38:49.610085 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:49.609969    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 10:38:49.610142 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:49.610093    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jpwk2\""
Sep 09 10:38:49.618548 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:49.618527    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-f54rk"]
Sep 09 10:38:49.672440 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:49.672414    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wphb\" (UniqueName: \"kubernetes.io/projected/e17d4ac4-8ae3-4cee-b448-bc97ee4e57ba-kube-api-access-5wphb\") pod \"authorino-operator-76b478bd78-f54rk\" (UID: \"e17d4ac4-8ae3-4cee-b448-bc97ee4e57ba\") " pod="kuadrant-system/authorino-operator-76b478bd78-f54rk"
Sep 09 10:38:49.773497 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:49.773460    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wphb\" (UniqueName: \"kubernetes.io/projected/e17d4ac4-8ae3-4cee-b448-bc97ee4e57ba-kube-api-access-5wphb\") pod \"authorino-operator-76b478bd78-f54rk\" (UID: \"e17d4ac4-8ae3-4cee-b448-bc97ee4e57ba\") " pod="kuadrant-system/authorino-operator-76b478bd78-f54rk"
Sep 09 10:38:49.786157 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:49.786136    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wphb\" (UniqueName: \"kubernetes.io/projected/e17d4ac4-8ae3-4cee-b448-bc97ee4e57ba-kube-api-access-5wphb\") pod \"authorino-operator-76b478bd78-f54rk\" (UID: \"e17d4ac4-8ae3-4cee-b448-bc97ee4e57ba\") " pod="kuadrant-system/authorino-operator-76b478bd78-f54rk"
Sep 09 10:38:49.916744 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:49.916678    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-f54rk"
Sep 09 10:38:50.192291 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:50.192267    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-f54rk"]
Sep 09 10:38:50.193640 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:38:50.193614    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17d4ac4_8ae3_4cee_b448_bc97ee4e57ba.slice/crio-ff458bc1867e9c7fddd4449f42ce1a5bafe892d05e31fc90f226487161214f34 WatchSource:0}: Error finding container ff458bc1867e9c7fddd4449f42ce1a5bafe892d05e31fc90f226487161214f34: Status 404 returned error can't find the container with id ff458bc1867e9c7fddd4449f42ce1a5bafe892d05e31fc90f226487161214f34
Sep 09 10:38:50.326912 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:50.326880    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:50.508008 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:50.507979    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-f54rk" event={"ID":"e17d4ac4-8ae3-4cee-b448-bc97ee4e57ba","Type":"ContainerStarted","Data":"ff458bc1867e9c7fddd4449f42ce1a5bafe892d05e31fc90f226487161214f34"}
Sep 09 10:38:50.576819 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:50.576788    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:51.833083 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:51.833055    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5kmnp"]
Sep 09 10:38:51.837303 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:51.837286    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5kmnp"
Sep 09 10:38:51.840183 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:51.840160    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8snd6\""
Sep 09 10:38:51.844004 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:51.843897    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 09 10:38:51.856965 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:51.856945    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5kmnp"]
Sep 09 10:38:51.889436 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:51.889402    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r65vx\" (UniqueName: \"kubernetes.io/projected/4404cd16-87dd-4f1a-9ba6-65fa0a3cc57f-kube-api-access-r65vx\") pod \"dns-operator-controller-manager-7b84cd7c9c-5kmnp\" (UID: \"4404cd16-87dd-4f1a-9ba6-65fa0a3cc57f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5kmnp"
Sep 09 10:38:51.990698 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:51.990666    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65vx\" (UniqueName: \"kubernetes.io/projected/4404cd16-87dd-4f1a-9ba6-65fa0a3cc57f-kube-api-access-r65vx\") pod \"dns-operator-controller-manager-7b84cd7c9c-5kmnp\" (UID: \"4404cd16-87dd-4f1a-9ba6-65fa0a3cc57f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5kmnp"
Sep 09 10:38:52.016959 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:52.016926    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65vx\" (UniqueName: \"kubernetes.io/projected/4404cd16-87dd-4f1a-9ba6-65fa0a3cc57f-kube-api-access-r65vx\") pod \"dns-operator-controller-manager-7b84cd7c9c-5kmnp\" (UID: \"4404cd16-87dd-4f1a-9ba6-65fa0a3cc57f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5kmnp"
Sep 09 10:38:52.044610 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:52.044577    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:52.146962 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:52.146888    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5kmnp"
Sep 09 10:38:52.418346 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:52.418284    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5kmnp"]
Sep 09 10:38:52.420275 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:38:52.420247    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4404cd16_87dd_4f1a_9ba6_65fa0a3cc57f.slice/crio-903498e71c4cf5ea31b875ef52ff2d6672671e4160c1d0c6296b5f544bc354b5 WatchSource:0}: Error finding container 903498e71c4cf5ea31b875ef52ff2d6672671e4160c1d0c6296b5f544bc354b5: Status 404 returned error can't find the container with id 903498e71c4cf5ea31b875ef52ff2d6672671e4160c1d0c6296b5f544bc354b5
Sep 09 10:38:52.518519 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:52.518479    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5kmnp" event={"ID":"4404cd16-87dd-4f1a-9ba6-65fa0a3cc57f","Type":"ContainerStarted","Data":"903498e71c4cf5ea31b875ef52ff2d6672671e4160c1d0c6296b5f544bc354b5"}
Sep 09 10:38:52.720381 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:52.720296    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:53.149867 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:53.149835    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:53.365676 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:53.365636    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:53.473124 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:53.473046    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:53.566420 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:53.566393    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:54.529874 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:54.529834    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-f54rk" event={"ID":"e17d4ac4-8ae3-4cee-b448-bc97ee4e57ba","Type":"ContainerStarted","Data":"8f1f9bb81c89473a6df66559d18e69dceed84540ce730446cfe799bf21c39949"}
Sep 09 10:38:54.530384 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:54.529921    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-f54rk"
Sep 09 10:38:54.554089 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:54.553935    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-f54rk" podStartSLOduration=1.707950235 podStartE2EDuration="5.553917417s" totalImagesPullingTime="3.845967182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:38:54.55351466 +0000 UTC m=+774.217153634" watchObservedRunningTime="2026-09-09 10:38:54.553917417 +0000 UTC m=+774.217556392"
Sep 09 10:38:55.538116 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:55.538078    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5kmnp" event={"ID":"4404cd16-87dd-4f1a-9ba6-65fa0a3cc57f","Type":"ContainerStarted","Data":"fac7e0b9d7019994566ee0b8fb2498591cfea33e0398637a703dae0ed2396a00"}
Sep 09 10:38:55.561494 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:55.561452    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5kmnp" podStartSLOduration=2.017077602 podStartE2EDuration="4.561436644s" totalImagesPullingTime="2.544359042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:38:55.56037317 +0000 UTC m=+775.224012144" watchObservedRunningTime="2026-09-09 10:38:55.561436644 +0000 UTC m=+775.225075617"
Sep 09 10:38:56.477615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:56.477585    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"]
Sep 09 10:38:56.480599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:56.480585    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:38:56.484192 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:56.484169    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x9wq2\""
Sep 09 10:38:56.502132 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:56.502111    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"]
Sep 09 10:38:56.527119 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:56.527085    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbdde77f-3f7b-483f-b61f-80be79b031b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qn6qr\" (UID: \"fbdde77f-3f7b-483f-b61f-80be79b031b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:38:56.527244 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:56.527140    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcqs\" (UniqueName: \"kubernetes.io/projected/fbdde77f-3f7b-483f-b61f-80be79b031b2-kube-api-access-zhcqs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qn6qr\" (UID: \"fbdde77f-3f7b-483f-b61f-80be79b031b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:38:56.542584 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:56.542554    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5kmnp"
Sep 09 10:38:56.628258 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:56.628228    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbdde77f-3f7b-483f-b61f-80be79b031b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qn6qr\" (UID: \"fbdde77f-3f7b-483f-b61f-80be79b031b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:38:56.628414 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:56.628351    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcqs\" (UniqueName: \"kubernetes.io/projected/fbdde77f-3f7b-483f-b61f-80be79b031b2-kube-api-access-zhcqs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qn6qr\" (UID: \"fbdde77f-3f7b-483f-b61f-80be79b031b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:38:56.628605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:56.628586    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbdde77f-3f7b-483f-b61f-80be79b031b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qn6qr\" (UID: \"fbdde77f-3f7b-483f-b61f-80be79b031b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:38:56.638380 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:56.638362    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcqs\" (UniqueName: \"kubernetes.io/projected/fbdde77f-3f7b-483f-b61f-80be79b031b2-kube-api-access-zhcqs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qn6qr\" (UID: \"fbdde77f-3f7b-483f-b61f-80be79b031b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:38:56.791098 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:56.791068    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:38:56.922276 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:56.922239    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:57.070690 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:57.070669    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"]
Sep 09 10:38:57.071972 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:38:57.071947    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdde77f_3f7b_483f_b61f_80be79b031b2.slice/crio-cadaaec7bdcf4c006c8e4db0805112f90d00068d6191014169de796e32ec2f36 WatchSource:0}: Error finding container cadaaec7bdcf4c006c8e4db0805112f90d00068d6191014169de796e32ec2f36: Status 404 returned error can't find the container with id cadaaec7bdcf4c006c8e4db0805112f90d00068d6191014169de796e32ec2f36
Sep 09 10:38:57.548303 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:57.548271    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr" event={"ID":"fbdde77f-3f7b-483f-b61f-80be79b031b2","Type":"ContainerStarted","Data":"cadaaec7bdcf4c006c8e4db0805112f90d00068d6191014169de796e32ec2f36"}
Sep 09 10:38:57.971750 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:57.971676    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:38:59.958472 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:38:59.958442    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:00.058986 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:00.058953    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:00.313578 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:00.313544    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:01.904081 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:01.904042    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:02.571534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:02.571499    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr" event={"ID":"fbdde77f-3f7b-483f-b61f-80be79b031b2","Type":"ContainerStarted","Data":"bc5f590a48a9132dfcfaa11f1459466a26f6724fc05dfbaac917eb820f332894"}
Sep 09 10:39:02.571730 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:02.571632    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:39:02.596748 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:02.596693    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr" podStartSLOduration=1.236993112 podStartE2EDuration="6.596678083s" totalImagesPullingTime="5.359684971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:39:02.595194681 +0000 UTC m=+782.258833655" watchObservedRunningTime="2026-09-09 10:39:02.596678083 +0000 UTC m=+782.260317057"
Sep 09 10:39:03.520215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:03.520169    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:03.591118 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:03.591088    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:05.134400 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.134359    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:05.134902 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.134881    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"]
Sep 09 10:39:05.138155 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.138141    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"
Sep 09 10:39:05.140811 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.140791    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 10:39:05.140919 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.140838    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m88d8\""
Sep 09 10:39:05.140919 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.140838    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 10:39:05.147840 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.147819    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"]
Sep 09 10:39:05.201059 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.201033    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0aca6679-454a-4d6d-9f65-e4deda62852c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-j2lmx\" (UID: \"0aca6679-454a-4d6d-9f65-e4deda62852c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"
Sep 09 10:39:05.201235 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.201096    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0aca6679-454a-4d6d-9f65-e4deda62852c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-j2lmx\" (UID: \"0aca6679-454a-4d6d-9f65-e4deda62852c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"
Sep 09 10:39:05.201235 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.201116    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w2wl\" (UniqueName: \"kubernetes.io/projected/0aca6679-454a-4d6d-9f65-e4deda62852c-kube-api-access-7w2wl\") pod \"kuadrant-console-plugin-fbdd4f445-j2lmx\" (UID: \"0aca6679-454a-4d6d-9f65-e4deda62852c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"
Sep 09 10:39:05.302033 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.302002    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w2wl\" (UniqueName: \"kubernetes.io/projected/0aca6679-454a-4d6d-9f65-e4deda62852c-kube-api-access-7w2wl\") pod \"kuadrant-console-plugin-fbdd4f445-j2lmx\" (UID: \"0aca6679-454a-4d6d-9f65-e4deda62852c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"
Sep 09 10:39:05.302191 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.302050    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0aca6679-454a-4d6d-9f65-e4deda62852c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-j2lmx\" (UID: \"0aca6679-454a-4d6d-9f65-e4deda62852c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"
Sep 09 10:39:05.302191 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.302107    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0aca6679-454a-4d6d-9f65-e4deda62852c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-j2lmx\" (UID: \"0aca6679-454a-4d6d-9f65-e4deda62852c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"
Sep 09 10:39:05.302261 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:05.302217    2570 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 09 10:39:05.302322 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:05.302312    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0aca6679-454a-4d6d-9f65-e4deda62852c-plugin-serving-cert podName:0aca6679-454a-4d6d-9f65-e4deda62852c nodeName:}" failed. No retries permitted until 2026-09-09 10:39:05.802286497 +0000 UTC m=+785.465925449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/0aca6679-454a-4d6d-9f65-e4deda62852c-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-j2lmx" (UID: "0aca6679-454a-4d6d-9f65-e4deda62852c") : secret "plugin-serving-cert" not found
Sep 09 10:39:05.302744 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.302727    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0aca6679-454a-4d6d-9f65-e4deda62852c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-j2lmx\" (UID: \"0aca6679-454a-4d6d-9f65-e4deda62852c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"
Sep 09 10:39:05.314392 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.314369    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w2wl\" (UniqueName: \"kubernetes.io/projected/0aca6679-454a-4d6d-9f65-e4deda62852c-kube-api-access-7w2wl\") pod \"kuadrant-console-plugin-fbdd4f445-j2lmx\" (UID: \"0aca6679-454a-4d6d-9f65-e4deda62852c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"
Sep 09 10:39:05.418467 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.418391    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:05.541000 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.540974    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-f54rk"
Sep 09 10:39:05.806799 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.806744    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0aca6679-454a-4d6d-9f65-e4deda62852c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-j2lmx\" (UID: \"0aca6679-454a-4d6d-9f65-e4deda62852c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"
Sep 09 10:39:05.809256 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:05.809234    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0aca6679-454a-4d6d-9f65-e4deda62852c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-j2lmx\" (UID: \"0aca6679-454a-4d6d-9f65-e4deda62852c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"
Sep 09 10:39:06.059298 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:06.059212    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"
Sep 09 10:39:06.328079 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:06.328055    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx"]
Sep 09 10:39:06.329988 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:39:06.329958    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0aca6679_454a_4d6d_9f65_e4deda62852c.slice/crio-0f1fc3ace9b06c07c0bcf6365317078689a63b4ab1db74ca3f698acd738aa221 WatchSource:0}: Error finding container 0f1fc3ace9b06c07c0bcf6365317078689a63b4ab1db74ca3f698acd738aa221: Status 404 returned error can't find the container with id 0f1fc3ace9b06c07c0bcf6365317078689a63b4ab1db74ca3f698acd738aa221
Sep 09 10:39:06.591432 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:06.591340    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx" event={"ID":"0aca6679-454a-4d6d-9f65-e4deda62852c","Type":"ContainerStarted","Data":"0f1fc3ace9b06c07c0bcf6365317078689a63b4ab1db74ca3f698acd738aa221"}
Sep 09 10:39:07.550580 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:07.550554    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5kmnp"
Sep 09 10:39:08.643395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:08.643358    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:08.769592 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:08.769555    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:08.978254 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:08.978189    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:11.614736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:11.614698    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx" event={"ID":"0aca6679-454a-4d6d-9f65-e4deda62852c","Type":"ContainerStarted","Data":"d94b2ca2c47908f6cc4785906a173b39c7972046ca14e93f46ea2cd169637cd4"}
Sep 09 10:39:11.635827 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:11.635782    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j2lmx" podStartSLOduration=1.576920575 podStartE2EDuration="6.635750106s" totalImagesPullingTime="5.058829531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:39:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:39:11.634879005 +0000 UTC m=+791.298517981" watchObservedRunningTime="2026-09-09 10:39:11.635750106 +0000 UTC m=+791.299389077"
Sep 09 10:39:13.578192 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:13.578157    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:39:15.435791 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.435735    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"]
Sep 09 10:39:15.439120 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.439103    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:15.468146 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.468117    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"]
Sep 09 10:39:15.595859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.595757    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6jfr\" (UniqueName: \"kubernetes.io/projected/e955f371-450e-4c0a-b170-ca9193ff4c56-kube-api-access-p6jfr\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-82q4m\" (UID: \"e955f371-450e-4c0a-b170-ca9193ff4c56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:15.595859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.595843    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e955f371-450e-4c0a-b170-ca9193ff4c56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-82q4m\" (UID: \"e955f371-450e-4c0a-b170-ca9193ff4c56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:15.697030 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.696944    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e955f371-450e-4c0a-b170-ca9193ff4c56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-82q4m\" (UID: \"e955f371-450e-4c0a-b170-ca9193ff4c56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:15.697030 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.697000    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6jfr\" (UniqueName: \"kubernetes.io/projected/e955f371-450e-4c0a-b170-ca9193ff4c56-kube-api-access-p6jfr\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-82q4m\" (UID: \"e955f371-450e-4c0a-b170-ca9193ff4c56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:15.697376 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.697353    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e955f371-450e-4c0a-b170-ca9193ff4c56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-82q4m\" (UID: \"e955f371-450e-4c0a-b170-ca9193ff4c56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:15.712185 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.712153    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6jfr\" (UniqueName: \"kubernetes.io/projected/e955f371-450e-4c0a-b170-ca9193ff4c56-kube-api-access-p6jfr\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-82q4m\" (UID: \"e955f371-450e-4c0a-b170-ca9193ff4c56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:15.725032 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.725003    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:15.749168 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.749141    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:15.833840 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.833807    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"]
Sep 09 10:39:15.834012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.833991    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr" podUID="fbdde77f-3f7b-483f-b61f-80be79b031b2" containerName="manager" containerID="cri-o://bc5f590a48a9132dfcfaa11f1459466a26f6724fc05dfbaac917eb820f332894" gracePeriod=2
Sep 09 10:39:15.848968 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.848906    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:15.849090 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.848992    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"]
Sep 09 10:39:15.856476 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:15.856444    2570 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"fbdde77f-3f7b-483f-b61f-80be79b031b2\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-qn6qr\": pods \"kuadrant-operator-controller-manager-76f8f686bf-qn6qr\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:39:15.872177 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.872152    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"]
Sep 09 10:39:15.879635 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.879610    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"]
Sep 09 10:39:15.880044 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.880028    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbdde77f-3f7b-483f-b61f-80be79b031b2" containerName="manager"
Sep 09 10:39:15.880098 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.880048    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbdde77f-3f7b-483f-b61f-80be79b031b2" containerName="manager"
Sep 09 10:39:15.880150 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.880140    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbdde77f-3f7b-483f-b61f-80be79b031b2" containerName="manager"
Sep 09 10:39:15.889950 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.889913    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"
Sep 09 10:39:15.891121 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.891098    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"]
Sep 09 10:39:15.900475 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.900441    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"]
Sep 09 10:39:15.910049 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.910029    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"]
Sep 09 10:39:15.913205 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.913191    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"
Sep 09 10:39:15.928210 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:15.928170    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qn6qr\" is forbidden: User \"system:node:ip-10-0-128-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-193.ec2.internal' and this object" podUID="fbdde77f-3f7b-483f-b61f-80be79b031b2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:39:15.930321 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:15.930291    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qn6qr\" is forbidden: User \"system:node:ip-10-0-128-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-193.ec2.internal' and this object" podUID="fbdde77f-3f7b-483f-b61f-80be79b031b2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:39:15.932301 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.932278    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"]
Sep 09 10:39:15.998938 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.998908    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rbg\" (UniqueName: \"kubernetes.io/projected/86d11a15-62e7-4f31-b66b-0b02b0596ff9-kube-api-access-v9rbg\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xgdf9\" (UID: \"86d11a15-62e7-4f31-b66b-0b02b0596ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"
Sep 09 10:39:15.999081 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.998964    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86d11a15-62e7-4f31-b66b-0b02b0596ff9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xgdf9\" (UID: \"86d11a15-62e7-4f31-b66b-0b02b0596ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"
Sep 09 10:39:15.999081 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.999033    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkbw\" (UniqueName: \"kubernetes.io/projected/5ed84077-79ad-49cc-8e4d-97c3ddc05e75-kube-api-access-zpkbw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pq7pv\" (UID: \"5ed84077-79ad-49cc-8e4d-97c3ddc05e75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"
Sep 09 10:39:15.999081 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:15.999077    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ed84077-79ad-49cc-8e4d-97c3ddc05e75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pq7pv\" (UID: \"5ed84077-79ad-49cc-8e4d-97c3ddc05e75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"
Sep 09 10:39:16.066933 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.066907    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:39:16.069551 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:16.069523    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qn6qr\" is forbidden: User \"system:node:ip-10-0-128-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-193.ec2.internal' and this object" podUID="fbdde77f-3f7b-483f-b61f-80be79b031b2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:39:16.099564 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.099521    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ed84077-79ad-49cc-8e4d-97c3ddc05e75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pq7pv\" (UID: \"5ed84077-79ad-49cc-8e4d-97c3ddc05e75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"
Sep 09 10:39:16.099740 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.099599    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rbg\" (UniqueName: \"kubernetes.io/projected/86d11a15-62e7-4f31-b66b-0b02b0596ff9-kube-api-access-v9rbg\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xgdf9\" (UID: \"86d11a15-62e7-4f31-b66b-0b02b0596ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"
Sep 09 10:39:16.099740 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.099652    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86d11a15-62e7-4f31-b66b-0b02b0596ff9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xgdf9\" (UID: \"86d11a15-62e7-4f31-b66b-0b02b0596ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"
Sep 09 10:39:16.099740 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.099704    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkbw\" (UniqueName: \"kubernetes.io/projected/5ed84077-79ad-49cc-8e4d-97c3ddc05e75-kube-api-access-zpkbw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pq7pv\" (UID: \"5ed84077-79ad-49cc-8e4d-97c3ddc05e75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"
Sep 09 10:39:16.099993 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.099971    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ed84077-79ad-49cc-8e4d-97c3ddc05e75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pq7pv\" (UID: \"5ed84077-79ad-49cc-8e4d-97c3ddc05e75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"
Sep 09 10:39:16.100054 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.100041    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86d11a15-62e7-4f31-b66b-0b02b0596ff9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xgdf9\" (UID: \"86d11a15-62e7-4f31-b66b-0b02b0596ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"
Sep 09 10:39:16.115355 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.115318    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rbg\" (UniqueName: \"kubernetes.io/projected/86d11a15-62e7-4f31-b66b-0b02b0596ff9-kube-api-access-v9rbg\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xgdf9\" (UID: \"86d11a15-62e7-4f31-b66b-0b02b0596ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"
Sep 09 10:39:16.123860 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.123831    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkbw\" (UniqueName: \"kubernetes.io/projected/5ed84077-79ad-49cc-8e4d-97c3ddc05e75-kube-api-access-zpkbw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pq7pv\" (UID: \"5ed84077-79ad-49cc-8e4d-97c3ddc05e75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"
Sep 09 10:39:16.131288 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:16.131258    2570 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 09 10:39:16.131288 ip-10-0-128-193 kubenswrapper[2570]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-82q4m_kuadrant-system_e955f371-450e-4c0a-b170-ca9193ff4c56_0(882ac83a25fd6444e982915647007a7b2e0c2a984297a318fb67e5c39be4dea1): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-82q4m to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"882ac83a25fd6444e982915647007a7b2e0c2a984297a318fb67e5c39be4dea1" Netns:"/var/run/netns/b8ae1189-e204-4ca7-ad68-710f33108eab" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-82q4m;K8S_POD_INFRA_CONTAINER_ID=882ac83a25fd6444e982915647007a7b2e0c2a984297a318fb67e5c39be4dea1;K8S_POD_UID=e955f371-450e-4c0a-b170-ca9193ff4c56" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m/e955f371-450e-4c0a-b170-ca9193ff4c56:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m 882ac83a25fd6444e982915647007a7b2e0c2a984297a318fb67e5c39be4dea1 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:1a [10.134.0.26/23]
Sep 09 10:39:16.131288 ip-10-0-128-193 kubenswrapper[2570]:         '
Sep 09 10:39:16.131288 ip-10-0-128-193 kubenswrapper[2570]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 09 10:39:16.131288 ip-10-0-128-193 kubenswrapper[2570]:  >
Sep 09 10:39:16.131465 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:16.131311    2570 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 09 10:39:16.131465 ip-10-0-128-193 kubenswrapper[2570]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-82q4m_kuadrant-system_e955f371-450e-4c0a-b170-ca9193ff4c56_0(882ac83a25fd6444e982915647007a7b2e0c2a984297a318fb67e5c39be4dea1): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-82q4m to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"882ac83a25fd6444e982915647007a7b2e0c2a984297a318fb67e5c39be4dea1" Netns:"/var/run/netns/b8ae1189-e204-4ca7-ad68-710f33108eab" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-82q4m;K8S_POD_INFRA_CONTAINER_ID=882ac83a25fd6444e982915647007a7b2e0c2a984297a318fb67e5c39be4dea1;K8S_POD_UID=e955f371-450e-4c0a-b170-ca9193ff4c56" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m/e955f371-450e-4c0a-b170-ca9193ff4c56:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m 882ac83a25fd6444e982915647007a7b2e0c2a984297a318fb67e5c39be4dea1 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:1a [10.134.0.26/23]
Sep 09 10:39:16.131465 ip-10-0-128-193 kubenswrapper[2570]:         '
Sep 09 10:39:16.131465 ip-10-0-128-193 kubenswrapper[2570]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 09 10:39:16.131465 ip-10-0-128-193 kubenswrapper[2570]:  > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:16.201047 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.201014    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhcqs\" (UniqueName: \"kubernetes.io/projected/fbdde77f-3f7b-483f-b61f-80be79b031b2-kube-api-access-zhcqs\") pod \"fbdde77f-3f7b-483f-b61f-80be79b031b2\" (UID: \"fbdde77f-3f7b-483f-b61f-80be79b031b2\") "
Sep 09 10:39:16.201202 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.201073    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbdde77f-3f7b-483f-b61f-80be79b031b2-extensions-socket-volume\") pod \"fbdde77f-3f7b-483f-b61f-80be79b031b2\" (UID: \"fbdde77f-3f7b-483f-b61f-80be79b031b2\") "
Sep 09 10:39:16.201534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.201508    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdde77f-3f7b-483f-b61f-80be79b031b2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fbdde77f-3f7b-483f-b61f-80be79b031b2" (UID: "fbdde77f-3f7b-483f-b61f-80be79b031b2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:39:16.203272 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.203213    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdde77f-3f7b-483f-b61f-80be79b031b2-kube-api-access-zhcqs" (OuterVolumeSpecName: "kube-api-access-zhcqs") pod "fbdde77f-3f7b-483f-b61f-80be79b031b2" (UID: "fbdde77f-3f7b-483f-b61f-80be79b031b2"). InnerVolumeSpecName "kube-api-access-zhcqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:39:16.208483 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.208467    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"
Sep 09 10:39:16.221663 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.221640    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"
Sep 09 10:39:16.302652 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.302619    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhcqs\" (UniqueName: \"kubernetes.io/projected/fbdde77f-3f7b-483f-b61f-80be79b031b2-kube-api-access-zhcqs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:39:16.302652 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.302654    2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbdde77f-3f7b-483f-b61f-80be79b031b2-extensions-socket-volume\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:39:16.496706 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.496680    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"]
Sep 09 10:39:16.497128 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:39:16.497107    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed84077_79ad_49cc_8e4d_97c3ddc05e75.slice/crio-b95e234009f433426253eb1ea37d60eae9698dd64c52cfd4af694d1f01a182d0 WatchSource:0}: Error finding container b95e234009f433426253eb1ea37d60eae9698dd64c52cfd4af694d1f01a182d0: Status 404 returned error can't find the container with id b95e234009f433426253eb1ea37d60eae9698dd64c52cfd4af694d1f01a182d0
Sep 09 10:39:16.515595 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.515570    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"]
Sep 09 10:39:16.517716 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:39:16.517688    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d11a15_62e7_4f31_b66b_0b02b0596ff9.slice/crio-0a3a412cb862123c48bc5cddbc3eb54f82635a6e6ceced200227501642dab079 WatchSource:0}: Error finding container 0a3a412cb862123c48bc5cddbc3eb54f82635a6e6ceced200227501642dab079: Status 404 returned error can't find the container with id 0a3a412cb862123c48bc5cddbc3eb54f82635a6e6ceced200227501642dab079
Sep 09 10:39:16.638393 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.638358    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv" event={"ID":"5ed84077-79ad-49cc-8e4d-97c3ddc05e75","Type":"ContainerStarted","Data":"da208b32986bd3b5cb346a2cef129a6dc228d9cefff5b17d1fe200372b60e9ac"}
Sep 09 10:39:16.638592 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.638420    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv" event={"ID":"5ed84077-79ad-49cc-8e4d-97c3ddc05e75","Type":"ContainerStarted","Data":"b95e234009f433426253eb1ea37d60eae9698dd64c52cfd4af694d1f01a182d0"}
Sep 09 10:39:16.638592 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.638524    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"
Sep 09 10:39:16.639674 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.639650    2570 generic.go:356] "Generic (PLEG): container finished" podID="fbdde77f-3f7b-483f-b61f-80be79b031b2" containerID="bc5f590a48a9132dfcfaa11f1459466a26f6724fc05dfbaac917eb820f332894" exitCode=0
Sep 09 10:39:16.639753 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.639682    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:39:16.640010 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.639753    2570 scope.go:117] "RemoveContainer" containerID="bc5f590a48a9132dfcfaa11f1459466a26f6724fc05dfbaac917eb820f332894"
Sep 09 10:39:16.641848 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.641826    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9" event={"ID":"86d11a15-62e7-4f31-b66b-0b02b0596ff9","Type":"ContainerStarted","Data":"867691227abb2677ff2b4bf4c0046ed5d53387672e59bf0424e207a8e1c55478"}
Sep 09 10:39:16.641848 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.641843    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:16.641985 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.641872    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9" event={"ID":"86d11a15-62e7-4f31-b66b-0b02b0596ff9","Type":"ContainerStarted","Data":"0a3a412cb862123c48bc5cddbc3eb54f82635a6e6ceced200227501642dab079"}
Sep 09 10:39:16.642042 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.641991    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"
Sep 09 10:39:16.650513 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.650496    2570 scope.go:117] "RemoveContainer" containerID="bc5f590a48a9132dfcfaa11f1459466a26f6724fc05dfbaac917eb820f332894"
Sep 09 10:39:16.650802 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:16.650759    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5f590a48a9132dfcfaa11f1459466a26f6724fc05dfbaac917eb820f332894\": container with ID starting with bc5f590a48a9132dfcfaa11f1459466a26f6724fc05dfbaac917eb820f332894 not found: ID does not exist" containerID="bc5f590a48a9132dfcfaa11f1459466a26f6724fc05dfbaac917eb820f332894"
Sep 09 10:39:16.650902 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.650801    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5f590a48a9132dfcfaa11f1459466a26f6724fc05dfbaac917eb820f332894"} err="failed to get container status \"bc5f590a48a9132dfcfaa11f1459466a26f6724fc05dfbaac917eb820f332894\": rpc error: code = NotFound desc = could not find container \"bc5f590a48a9132dfcfaa11f1459466a26f6724fc05dfbaac917eb820f332894\": container with ID starting with bc5f590a48a9132dfcfaa11f1459466a26f6724fc05dfbaac917eb820f332894 not found: ID does not exist"
Sep 09 10:39:16.650993 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.650978    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:16.666747 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.666704    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv" podStartSLOduration=1.666690386 podStartE2EDuration="1.666690386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:39:16.666117489 +0000 UTC m=+796.329756464" watchObservedRunningTime="2026-09-09 10:39:16.666690386 +0000 UTC m=+796.330329360"
Sep 09 10:39:16.668526 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:16.668503    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qn6qr\" is forbidden: User \"system:node:ip-10-0-128-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-193.ec2.internal' and this object" podUID="fbdde77f-3f7b-483f-b61f-80be79b031b2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:39:16.703676 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:16.703640    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-82q4m\" is forbidden: User \"system:node:ip-10-0-128-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-193.ec2.internal' and this object" podUID="e955f371-450e-4c0a-b170-ca9193ff4c56" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:16.707311 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:16.707288    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qn6qr\" is forbidden: User \"system:node:ip-10-0-128-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-193.ec2.internal' and this object" podUID="fbdde77f-3f7b-483f-b61f-80be79b031b2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qn6qr"
Sep 09 10:39:16.805883 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.805796    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6jfr\" (UniqueName: \"kubernetes.io/projected/e955f371-450e-4c0a-b170-ca9193ff4c56-kube-api-access-p6jfr\") pod \"e955f371-450e-4c0a-b170-ca9193ff4c56\" (UID: \"e955f371-450e-4c0a-b170-ca9193ff4c56\") "
Sep 09 10:39:16.805883 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.805875    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e955f371-450e-4c0a-b170-ca9193ff4c56-extensions-socket-volume\") pod \"e955f371-450e-4c0a-b170-ca9193ff4c56\" (UID: \"e955f371-450e-4c0a-b170-ca9193ff4c56\") "
Sep 09 10:39:16.806141 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.806116    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e955f371-450e-4c0a-b170-ca9193ff4c56-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e955f371-450e-4c0a-b170-ca9193ff4c56" (UID: "e955f371-450e-4c0a-b170-ca9193ff4c56"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:39:16.808103 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.808084    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e955f371-450e-4c0a-b170-ca9193ff4c56-kube-api-access-p6jfr" (OuterVolumeSpecName: "kube-api-access-p6jfr") pod "e955f371-450e-4c0a-b170-ca9193ff4c56" (UID: "e955f371-450e-4c0a-b170-ca9193ff4c56"). InnerVolumeSpecName "kube-api-access-p6jfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:39:16.838618 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.838589    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e955f371-450e-4c0a-b170-ca9193ff4c56" path="/var/lib/kubelet/pods/e955f371-450e-4c0a-b170-ca9193ff4c56/volumes"
Sep 09 10:39:16.838865 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.838852    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbdde77f-3f7b-483f-b61f-80be79b031b2" path="/var/lib/kubelet/pods/fbdde77f-3f7b-483f-b61f-80be79b031b2/volumes"
Sep 09 10:39:16.907408 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.907378    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6jfr\" (UniqueName: \"kubernetes.io/projected/e955f371-450e-4c0a-b170-ca9193ff4c56-kube-api-access-p6jfr\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:39:16.907408 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:16.907403    2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e955f371-450e-4c0a-b170-ca9193ff4c56-extensions-socket-volume\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:39:17.649232 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:17.649196    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:17.654521 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:17.654487    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-82q4m\" is forbidden: User \"system:node:ip-10-0-128-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-193.ec2.internal' and this object" podUID="e955f371-450e-4c0a-b170-ca9193ff4c56" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-82q4m"
Sep 09 10:39:18.821242 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:18.821208    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:18.926111 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:18.926069    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:22.313350 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:22.313319    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:22.379694 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:22.379661    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:27.651699 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:27.651618    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"
Sep 09 10:39:27.651699 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:27.651691    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"
Sep 09 10:39:27.676493 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:27.676446    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9" podStartSLOduration=12.676432634 podStartE2EDuration="12.676432634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:39:16.700633584 +0000 UTC m=+796.364272557" watchObservedRunningTime="2026-09-09 10:39:27.676432634 +0000 UTC m=+807.340071609"
Sep 09 10:39:27.736075 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:27.736043    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"]
Sep 09 10:39:27.736250 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:27.736226    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv" podUID="5ed84077-79ad-49cc-8e4d-97c3ddc05e75" containerName="manager" containerID="cri-o://da208b32986bd3b5cb346a2cef129a6dc228d9cefff5b17d1fe200372b60e9ac" gracePeriod=10
Sep 09 10:39:27.984974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:27.984954    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"
Sep 09 10:39:28.099032 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.099004    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpkbw\" (UniqueName: \"kubernetes.io/projected/5ed84077-79ad-49cc-8e4d-97c3ddc05e75-kube-api-access-zpkbw\") pod \"5ed84077-79ad-49cc-8e4d-97c3ddc05e75\" (UID: \"5ed84077-79ad-49cc-8e4d-97c3ddc05e75\") "
Sep 09 10:39:28.099207 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.099071    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ed84077-79ad-49cc-8e4d-97c3ddc05e75-extensions-socket-volume\") pod \"5ed84077-79ad-49cc-8e4d-97c3ddc05e75\" (UID: \"5ed84077-79ad-49cc-8e4d-97c3ddc05e75\") "
Sep 09 10:39:28.099431 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.099407    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed84077-79ad-49cc-8e4d-97c3ddc05e75-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5ed84077-79ad-49cc-8e4d-97c3ddc05e75" (UID: "5ed84077-79ad-49cc-8e4d-97c3ddc05e75"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:39:28.100954 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.100935    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed84077-79ad-49cc-8e4d-97c3ddc05e75-kube-api-access-zpkbw" (OuterVolumeSpecName: "kube-api-access-zpkbw") pod "5ed84077-79ad-49cc-8e4d-97c3ddc05e75" (UID: "5ed84077-79ad-49cc-8e4d-97c3ddc05e75"). InnerVolumeSpecName "kube-api-access-zpkbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:39:28.147817 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.147755    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:28.200107 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.200080    2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ed84077-79ad-49cc-8e4d-97c3ddc05e75-extensions-socket-volume\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:39:28.200107 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.200101    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpkbw\" (UniqueName: \"kubernetes.io/projected/5ed84077-79ad-49cc-8e4d-97c3ddc05e75-kube-api-access-zpkbw\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:39:28.245605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.245581    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:28.358911 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.358882    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:28.700616 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.700536    2570 generic.go:356] "Generic (PLEG): container finished" podID="5ed84077-79ad-49cc-8e4d-97c3ddc05e75" containerID="da208b32986bd3b5cb346a2cef129a6dc228d9cefff5b17d1fe200372b60e9ac" exitCode=0
Sep 09 10:39:28.700616 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.700573    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"
Sep 09 10:39:28.700616 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.700608    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv" event={"ID":"5ed84077-79ad-49cc-8e4d-97c3ddc05e75","Type":"ContainerDied","Data":"da208b32986bd3b5cb346a2cef129a6dc228d9cefff5b17d1fe200372b60e9ac"}
Sep 09 10:39:28.701140 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.700655    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv" event={"ID":"5ed84077-79ad-49cc-8e4d-97c3ddc05e75","Type":"ContainerDied","Data":"b95e234009f433426253eb1ea37d60eae9698dd64c52cfd4af694d1f01a182d0"}
Sep 09 10:39:28.701140 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.700679    2570 scope.go:117] "RemoveContainer" containerID="da208b32986bd3b5cb346a2cef129a6dc228d9cefff5b17d1fe200372b60e9ac"
Sep 09 10:39:28.708658 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.708645    2570 scope.go:117] "RemoveContainer" containerID="da208b32986bd3b5cb346a2cef129a6dc228d9cefff5b17d1fe200372b60e9ac"
Sep 09 10:39:28.708940 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:28.708922    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da208b32986bd3b5cb346a2cef129a6dc228d9cefff5b17d1fe200372b60e9ac\": container with ID starting with da208b32986bd3b5cb346a2cef129a6dc228d9cefff5b17d1fe200372b60e9ac not found: ID does not exist" containerID="da208b32986bd3b5cb346a2cef129a6dc228d9cefff5b17d1fe200372b60e9ac"
Sep 09 10:39:28.708995 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.708950    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da208b32986bd3b5cb346a2cef129a6dc228d9cefff5b17d1fe200372b60e9ac"} err="failed to get container status \"da208b32986bd3b5cb346a2cef129a6dc228d9cefff5b17d1fe200372b60e9ac\": rpc error: code = NotFound desc = could not find container \"da208b32986bd3b5cb346a2cef129a6dc228d9cefff5b17d1fe200372b60e9ac\": container with ID starting with da208b32986bd3b5cb346a2cef129a6dc228d9cefff5b17d1fe200372b60e9ac not found: ID does not exist"
Sep 09 10:39:28.736305 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.736282    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"]
Sep 09 10:39:28.739247 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.739225    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pq7pv"]
Sep 09 10:39:28.838656 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:28.838618    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ed84077-79ad-49cc-8e4d-97c3ddc05e75" path="/var/lib/kubelet/pods/5ed84077-79ad-49cc-8e4d-97c3ddc05e75/volumes"
Sep 09 10:39:29.515391 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:29.515361    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:30.714988 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:30.714953    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:35.270674 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:35.270642    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:35.415996 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:35.415962    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:44.061120 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.061089    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"]
Sep 09 10:39:44.061603 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.061401    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ed84077-79ad-49cc-8e4d-97c3ddc05e75" containerName="manager"
Sep 09 10:39:44.061603 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.061412    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ed84077-79ad-49cc-8e4d-97c3ddc05e75" containerName="manager"
Sep 09 10:39:44.061603 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.061464    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ed84077-79ad-49cc-8e4d-97c3ddc05e75" containerName="manager"
Sep 09 10:39:44.075388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.075367    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.078563 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.078540    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 10:39:44.078699 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.078542    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9grsq\""
Sep 09 10:39:44.078784 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.078745    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"]
Sep 09 10:39:44.090235 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.090214    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:44.118491 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.118455    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd184f91-b4c9-4291-9056-e6529b1f8541-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.118619 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.118502    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.118619 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.118577    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd184f91-b4c9-4291-9056-e6529b1f8541-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.118741 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.118630    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.118741 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.118711    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.118865 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.118739    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.118865 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.118830    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.118865 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.118857    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt5sq\" (UniqueName: \"kubernetes.io/projected/cd184f91-b4c9-4291-9056-e6529b1f8541-kube-api-access-nt5sq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.119007 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.118884    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd184f91-b4c9-4291-9056-e6529b1f8541-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.220231 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.220204    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd184f91-b4c9-4291-9056-e6529b1f8541-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.220231 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.220241    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.220466 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.220294    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.220466 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.220313    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.220466 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.220351    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.220466 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.220369    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt5sq\" (UniqueName: \"kubernetes.io/projected/cd184f91-b4c9-4291-9056-e6529b1f8541-kube-api-access-nt5sq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.220466 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.220388    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd184f91-b4c9-4291-9056-e6529b1f8541-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.220466 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.220426    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd184f91-b4c9-4291-9056-e6529b1f8541-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.220466 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.220457    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.220875 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.220637    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.220875 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.220654    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.220875 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.220834    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.221043 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.220949    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.221218 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.221196    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd184f91-b4c9-4291-9056-e6529b1f8541-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.222935 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.222913    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd184f91-b4c9-4291-9056-e6529b1f8541-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.223077 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.223059    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd184f91-b4c9-4291-9056-e6529b1f8541-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.228612 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.228593    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt5sq\" (UniqueName: \"kubernetes.io/projected/cd184f91-b4c9-4291-9056-e6529b1f8541-kube-api-access-nt5sq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.228693 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.228677    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd184f91-b4c9-4291-9056-e6529b1f8541-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lv97z\" (UID: \"cd184f91-b4c9-4291-9056-e6529b1f8541\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.231999 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.231976    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:39:44.387141 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.387065    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:44.477734 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.477665    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"]
Sep 09 10:39:44.477923 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.477881    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9" podUID="86d11a15-62e7-4f31-b66b-0b02b0596ff9" containerName="manager" containerID="cri-o://867691227abb2677ff2b4bf4c0046ed5d53387672e59bf0424e207a8e1c55478" gracePeriod=10
Sep 09 10:39:44.691027 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.691005    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"]
Sep 09 10:39:44.693542 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:39:44.693510    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd184f91_b4c9_4291_9056_e6529b1f8541.slice/crio-3301f65aad98b868838eaf0f244cf9203adbe47eeaa7c5c1dd4876b6db7c9ab8 WatchSource:0}: Error finding container 3301f65aad98b868838eaf0f244cf9203adbe47eeaa7c5c1dd4876b6db7c9ab8: Status 404 returned error can't find the container with id 3301f65aad98b868838eaf0f244cf9203adbe47eeaa7c5c1dd4876b6db7c9ab8
Sep 09 10:39:44.769315 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.769288    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"
Sep 09 10:39:44.770209 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.770182    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z" event={"ID":"cd184f91-b4c9-4291-9056-e6529b1f8541","Type":"ContainerStarted","Data":"3301f65aad98b868838eaf0f244cf9203adbe47eeaa7c5c1dd4876b6db7c9ab8"}
Sep 09 10:39:44.771536 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.771515    2570 generic.go:356] "Generic (PLEG): container finished" podID="86d11a15-62e7-4f31-b66b-0b02b0596ff9" containerID="867691227abb2677ff2b4bf4c0046ed5d53387672e59bf0424e207a8e1c55478" exitCode=0
Sep 09 10:39:44.771624 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.771542    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9" event={"ID":"86d11a15-62e7-4f31-b66b-0b02b0596ff9","Type":"ContainerDied","Data":"867691227abb2677ff2b4bf4c0046ed5d53387672e59bf0424e207a8e1c55478"}
Sep 09 10:39:44.771624 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.771591    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9" event={"ID":"86d11a15-62e7-4f31-b66b-0b02b0596ff9","Type":"ContainerDied","Data":"0a3a412cb862123c48bc5cddbc3eb54f82635a6e6ceced200227501642dab079"}
Sep 09 10:39:44.771624 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.771613    2570 scope.go:117] "RemoveContainer" containerID="867691227abb2677ff2b4bf4c0046ed5d53387672e59bf0424e207a8e1c55478"
Sep 09 10:39:44.771730 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.771555    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"
Sep 09 10:39:44.779421 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.779406    2570 scope.go:117] "RemoveContainer" containerID="867691227abb2677ff2b4bf4c0046ed5d53387672e59bf0424e207a8e1c55478"
Sep 09 10:39:44.779664 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:39:44.779648    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"867691227abb2677ff2b4bf4c0046ed5d53387672e59bf0424e207a8e1c55478\": container with ID starting with 867691227abb2677ff2b4bf4c0046ed5d53387672e59bf0424e207a8e1c55478 not found: ID does not exist" containerID="867691227abb2677ff2b4bf4c0046ed5d53387672e59bf0424e207a8e1c55478"
Sep 09 10:39:44.779726 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.779676    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"867691227abb2677ff2b4bf4c0046ed5d53387672e59bf0424e207a8e1c55478"} err="failed to get container status \"867691227abb2677ff2b4bf4c0046ed5d53387672e59bf0424e207a8e1c55478\": rpc error: code = NotFound desc = could not find container \"867691227abb2677ff2b4bf4c0046ed5d53387672e59bf0424e207a8e1c55478\": container with ID starting with 867691227abb2677ff2b4bf4c0046ed5d53387672e59bf0424e207a8e1c55478 not found: ID does not exist"
Sep 09 10:39:44.824736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.824696    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9rbg\" (UniqueName: \"kubernetes.io/projected/86d11a15-62e7-4f31-b66b-0b02b0596ff9-kube-api-access-v9rbg\") pod \"86d11a15-62e7-4f31-b66b-0b02b0596ff9\" (UID: \"86d11a15-62e7-4f31-b66b-0b02b0596ff9\") "
Sep 09 10:39:44.824896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.824749    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86d11a15-62e7-4f31-b66b-0b02b0596ff9-extensions-socket-volume\") pod \"86d11a15-62e7-4f31-b66b-0b02b0596ff9\" (UID: \"86d11a15-62e7-4f31-b66b-0b02b0596ff9\") "
Sep 09 10:39:44.825160 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.825131    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d11a15-62e7-4f31-b66b-0b02b0596ff9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "86d11a15-62e7-4f31-b66b-0b02b0596ff9" (UID: "86d11a15-62e7-4f31-b66b-0b02b0596ff9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:39:44.826775 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.826742    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d11a15-62e7-4f31-b66b-0b02b0596ff9-kube-api-access-v9rbg" (OuterVolumeSpecName: "kube-api-access-v9rbg") pod "86d11a15-62e7-4f31-b66b-0b02b0596ff9" (UID: "86d11a15-62e7-4f31-b66b-0b02b0596ff9"). InnerVolumeSpecName "kube-api-access-v9rbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:39:44.926131 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.926055    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9rbg\" (UniqueName: \"kubernetes.io/projected/86d11a15-62e7-4f31-b66b-0b02b0596ff9-kube-api-access-v9rbg\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:39:44.926131 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:44.926084    2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86d11a15-62e7-4f31-b66b-0b02b0596ff9-extensions-socket-volume\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:39:45.090246 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:45.090206    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"]
Sep 09 10:39:45.096469 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:45.096446    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgdf9"]
Sep 09 10:39:46.839117 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:46.839084    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86d11a15-62e7-4f31-b66b-0b02b0596ff9" path="/var/lib/kubelet/pods/86d11a15-62e7-4f31-b66b-0b02b0596ff9/volumes"
Sep 09 10:39:51.231746 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:51.231715    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:39:51.232006 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:51.231797    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:39:51.232006 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:51.231825    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:39:51.804945 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:51.804911    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z" event={"ID":"cd184f91-b4c9-4291-9056-e6529b1f8541","Type":"ContainerStarted","Data":"d63707c37670f4573172911380fa70e619fb68a14b143fca7d614fffdf2a08e1"}
Sep 09 10:39:51.827957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:51.827911    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z" podStartSLOduration=1.292269564 podStartE2EDuration="7.827896359s" totalImagesPullingTime="6.535626795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:39:51.826120385 +0000 UTC m=+831.489759359" watchObservedRunningTime="2026-09-09 10:39:51.827896359 +0000 UTC m=+831.491535385"
Sep 09 10:39:52.387566 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:52.387533    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:52.392048 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:52.392024    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:52.810267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:52.810236    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:39:52.811186 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:39:52.811167    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:40:02.498646 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.498605    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shsmf"]
Sep 09 10:40:02.499044 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.498985    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d11a15-62e7-4f31-b66b-0b02b0596ff9" containerName="manager"
Sep 09 10:40:02.499044 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.498999    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d11a15-62e7-4f31-b66b-0b02b0596ff9" containerName="manager"
Sep 09 10:40:02.499114 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.499057    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d11a15-62e7-4f31-b66b-0b02b0596ff9" containerName="manager"
Sep 09 10:40:02.503296 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.503279    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-shsmf"
Sep 09 10:40:02.506060 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.506039    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 10:40:02.512435 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.512412    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shsmf"]
Sep 09 10:40:02.562168 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.562127    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:02.572065 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.572034    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swdgf\" (UniqueName: \"kubernetes.io/projected/80572c9b-c043-4777-95e9-19dd97ac3574-kube-api-access-swdgf\") pod \"limitador-limitador-8588749977-shsmf\" (UID: \"80572c9b-c043-4777-95e9-19dd97ac3574\") " pod="kuadrant-system/limitador-limitador-8588749977-shsmf"
Sep 09 10:40:02.572200 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.572155    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80572c9b-c043-4777-95e9-19dd97ac3574-config-file\") pod \"limitador-limitador-8588749977-shsmf\" (UID: \"80572c9b-c043-4777-95e9-19dd97ac3574\") " pod="kuadrant-system/limitador-limitador-8588749977-shsmf"
Sep 09 10:40:02.592528 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.592500    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shsmf"]
Sep 09 10:40:02.658325 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.658297    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:02.673511 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.673490    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swdgf\" (UniqueName: \"kubernetes.io/projected/80572c9b-c043-4777-95e9-19dd97ac3574-kube-api-access-swdgf\") pod \"limitador-limitador-8588749977-shsmf\" (UID: \"80572c9b-c043-4777-95e9-19dd97ac3574\") " pod="kuadrant-system/limitador-limitador-8588749977-shsmf"
Sep 09 10:40:02.673600 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.673568    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80572c9b-c043-4777-95e9-19dd97ac3574-config-file\") pod \"limitador-limitador-8588749977-shsmf\" (UID: \"80572c9b-c043-4777-95e9-19dd97ac3574\") " pod="kuadrant-system/limitador-limitador-8588749977-shsmf"
Sep 09 10:40:02.674128 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.674112    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80572c9b-c043-4777-95e9-19dd97ac3574-config-file\") pod \"limitador-limitador-8588749977-shsmf\" (UID: \"80572c9b-c043-4777-95e9-19dd97ac3574\") " pod="kuadrant-system/limitador-limitador-8588749977-shsmf"
Sep 09 10:40:02.682932 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.682908    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swdgf\" (UniqueName: \"kubernetes.io/projected/80572c9b-c043-4777-95e9-19dd97ac3574-kube-api-access-swdgf\") pod \"limitador-limitador-8588749977-shsmf\" (UID: \"80572c9b-c043-4777-95e9-19dd97ac3574\") " pod="kuadrant-system/limitador-limitador-8588749977-shsmf"
Sep 09 10:40:02.813056 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:02.812969    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-shsmf"
Sep 09 10:40:03.086553 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.086529    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shsmf"]
Sep 09 10:40:03.088369 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:40:03.088343    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80572c9b_c043_4777_95e9_19dd97ac3574.slice/crio-b1576c6b3d3794eecd5c4a02d5a8f53ebe49cb3805a187ac75e03aedd110f8a7 WatchSource:0}: Error finding container b1576c6b3d3794eecd5c4a02d5a8f53ebe49cb3805a187ac75e03aedd110f8a7: Status 404 returned error can't find the container with id b1576c6b3d3794eecd5c4a02d5a8f53ebe49cb3805a187ac75e03aedd110f8a7
Sep 09 10:40:03.327912 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.327879    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jtgsc"]
Sep 09 10:40:03.332095 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.332079    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jtgsc"
Sep 09 10:40:03.335464 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.335444    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-s8k29\""
Sep 09 10:40:03.340964 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.340942    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jtgsc"]
Sep 09 10:40:03.366969 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.366944    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:03.379061 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.379039    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mght2\" (UniqueName: \"kubernetes.io/projected/b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e-kube-api-access-mght2\") pod \"authorino-7bcfd8f9cd-jtgsc\" (UID: \"b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jtgsc"
Sep 09 10:40:03.438799 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.438714    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:03.457462 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.457434    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-k988x"]
Sep 09 10:40:03.460597 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.460581    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-k988x"
Sep 09 10:40:03.470391 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.470371    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k988x"]
Sep 09 10:40:03.480217 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.480193    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mght2\" (UniqueName: \"kubernetes.io/projected/b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e-kube-api-access-mght2\") pod \"authorino-7bcfd8f9cd-jtgsc\" (UID: \"b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jtgsc"
Sep 09 10:40:03.490174 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.490150    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mght2\" (UniqueName: \"kubernetes.io/projected/b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e-kube-api-access-mght2\") pod \"authorino-7bcfd8f9cd-jtgsc\" (UID: \"b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jtgsc"
Sep 09 10:40:03.580854 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.580829    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v46md\" (UniqueName: \"kubernetes.io/projected/5ee36ca6-a3f4-496d-a71a-fa5dabf39f17-kube-api-access-v46md\") pod \"authorino-59c974bd48-k988x\" (UID: \"5ee36ca6-a3f4-496d-a71a-fa5dabf39f17\") " pod="kuadrant-system/authorino-59c974bd48-k988x"
Sep 09 10:40:03.642241 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.642178    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jtgsc"
Sep 09 10:40:03.682136 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.682106    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v46md\" (UniqueName: \"kubernetes.io/projected/5ee36ca6-a3f4-496d-a71a-fa5dabf39f17-kube-api-access-v46md\") pod \"authorino-59c974bd48-k988x\" (UID: \"5ee36ca6-a3f4-496d-a71a-fa5dabf39f17\") " pod="kuadrant-system/authorino-59c974bd48-k988x"
Sep 09 10:40:03.693803 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.693754    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v46md\" (UniqueName: \"kubernetes.io/projected/5ee36ca6-a3f4-496d-a71a-fa5dabf39f17-kube-api-access-v46md\") pod \"authorino-59c974bd48-k988x\" (UID: \"5ee36ca6-a3f4-496d-a71a-fa5dabf39f17\") " pod="kuadrant-system/authorino-59c974bd48-k988x"
Sep 09 10:40:03.769177 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.769145    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-k988x"
Sep 09 10:40:03.861089 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.861059    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-shsmf" event={"ID":"80572c9b-c043-4777-95e9-19dd97ac3574","Type":"ContainerStarted","Data":"b1576c6b3d3794eecd5c4a02d5a8f53ebe49cb3805a187ac75e03aedd110f8a7"}
Sep 09 10:40:03.908711 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:03.908619    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jtgsc"]
Sep 09 10:40:03.910902 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:40:03.910877    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb15c66d6_02b9_4bbc_b76d_3e9b0cf92f2e.slice/crio-f0386b75ccc590478e2ae3cbb1c038c209e73b700ad6914b5a9efa604188fa98 WatchSource:0}: Error finding container f0386b75ccc590478e2ae3cbb1c038c209e73b700ad6914b5a9efa604188fa98: Status 404 returned error can't find the container with id f0386b75ccc590478e2ae3cbb1c038c209e73b700ad6914b5a9efa604188fa98
Sep 09 10:40:04.037792 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:04.037755    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k988x"]
Sep 09 10:40:04.039974 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:40:04.039950    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ee36ca6_a3f4_496d_a71a_fa5dabf39f17.slice/crio-ff794047012ab2828240d3286ab871a6191d812809e8e50c405b1ed432409be8 WatchSource:0}: Error finding container ff794047012ab2828240d3286ab871a6191d812809e8e50c405b1ed432409be8: Status 404 returned error can't find the container with id ff794047012ab2828240d3286ab871a6191d812809e8e50c405b1ed432409be8
Sep 09 10:40:04.866179 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:04.866150    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-shsmf" event={"ID":"80572c9b-c043-4777-95e9-19dd97ac3574","Type":"ContainerStarted","Data":"ce5620a2996f609ef246d55bccd3e39af384de2a1884bd6103f60dfbf15e7613"}
Sep 09 10:40:04.866563 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:04.866225    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-shsmf"
Sep 09 10:40:04.867405 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:04.867381    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-k988x" event={"ID":"5ee36ca6-a3f4-496d-a71a-fa5dabf39f17","Type":"ContainerStarted","Data":"ff794047012ab2828240d3286ab871a6191d812809e8e50c405b1ed432409be8"}
Sep 09 10:40:04.868508 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:04.868488    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jtgsc" event={"ID":"b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e","Type":"ContainerStarted","Data":"f0386b75ccc590478e2ae3cbb1c038c209e73b700ad6914b5a9efa604188fa98"}
Sep 09 10:40:04.883780 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:04.883719    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-shsmf" podStartSLOduration=1.54485604 podStartE2EDuration="2.883705102s" totalImagesPullingTime="1.338849062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:40:04.881914798 +0000 UTC m=+844.545553763" watchObservedRunningTime="2026-09-09 10:40:04.883705102 +0000 UTC m=+844.547344073"
Sep 09 10:40:06.878565 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:06.878527    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-k988x" event={"ID":"5ee36ca6-a3f4-496d-a71a-fa5dabf39f17","Type":"ContainerStarted","Data":"a23a73a577dc1eb63755f1a96f1fe3d5b7e1e73d848932e1f1c9a24ba4e665cf"}
Sep 09 10:40:06.880195 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:06.880170    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jtgsc" event={"ID":"b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e","Type":"ContainerStarted","Data":"68056e05d47196d82ebb2ffd95833b4ddf2f40239def39ac7c206f2e3f9e6b52"}
Sep 09 10:40:06.894440 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:06.894396    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-k988x" podStartSLOduration=1.6539148799999999 podStartE2EDuration="3.894382352s" totalImagesPullingTime="2.240467472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:40:06.893015957 +0000 UTC m=+846.556654931" watchObservedRunningTime="2026-09-09 10:40:06.894382352 +0000 UTC m=+846.558021323"
Sep 09 10:40:06.908429 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:06.908387    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-jtgsc" podStartSLOduration=1.534850416 podStartE2EDuration="3.908376486s" totalImagesPullingTime="2.37352607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:40:06.907185959 +0000 UTC m=+846.570824934" watchObservedRunningTime="2026-09-09 10:40:06.908376486 +0000 UTC m=+846.572015460"
Sep 09 10:40:06.930842 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:06.930806    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jtgsc"]
Sep 09 10:40:08.889707 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:08.889648    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-jtgsc" podUID="b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e" containerName="authorino" containerID="cri-o://68056e05d47196d82ebb2ffd95833b4ddf2f40239def39ac7c206f2e3f9e6b52" gracePeriod=30
Sep 09 10:40:09.130593 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:09.130573    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jtgsc"
Sep 09 10:40:09.250356 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:09.250326    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mght2\" (UniqueName: \"kubernetes.io/projected/b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e-kube-api-access-mght2\") pod \"b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e\" (UID: \"b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e\") "
Sep 09 10:40:09.252290 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:09.252267    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e-kube-api-access-mght2" (OuterVolumeSpecName: "kube-api-access-mght2") pod "b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e" (UID: "b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e"). InnerVolumeSpecName "kube-api-access-mght2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:40:09.351067 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:09.351025    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mght2\" (UniqueName: \"kubernetes.io/projected/b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e-kube-api-access-mght2\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:40:09.895186 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:09.895152    2570 generic.go:356] "Generic (PLEG): container finished" podID="b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e" containerID="68056e05d47196d82ebb2ffd95833b4ddf2f40239def39ac7c206f2e3f9e6b52" exitCode=0
Sep 09 10:40:09.895589 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:09.895196    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jtgsc"
Sep 09 10:40:09.895589 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:09.895201    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jtgsc" event={"ID":"b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e","Type":"ContainerDied","Data":"68056e05d47196d82ebb2ffd95833b4ddf2f40239def39ac7c206f2e3f9e6b52"}
Sep 09 10:40:09.895589 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:09.895237    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jtgsc" event={"ID":"b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e","Type":"ContainerDied","Data":"f0386b75ccc590478e2ae3cbb1c038c209e73b700ad6914b5a9efa604188fa98"}
Sep 09 10:40:09.895589 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:09.895258    2570 scope.go:117] "RemoveContainer" containerID="68056e05d47196d82ebb2ffd95833b4ddf2f40239def39ac7c206f2e3f9e6b52"
Sep 09 10:40:09.903791 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:09.903757    2570 scope.go:117] "RemoveContainer" containerID="68056e05d47196d82ebb2ffd95833b4ddf2f40239def39ac7c206f2e3f9e6b52"
Sep 09 10:40:09.904052 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:40:09.904033    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68056e05d47196d82ebb2ffd95833b4ddf2f40239def39ac7c206f2e3f9e6b52\": container with ID starting with 68056e05d47196d82ebb2ffd95833b4ddf2f40239def39ac7c206f2e3f9e6b52 not found: ID does not exist" containerID="68056e05d47196d82ebb2ffd95833b4ddf2f40239def39ac7c206f2e3f9e6b52"
Sep 09 10:40:09.904116 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:09.904060    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68056e05d47196d82ebb2ffd95833b4ddf2f40239def39ac7c206f2e3f9e6b52"} err="failed to get container status \"68056e05d47196d82ebb2ffd95833b4ddf2f40239def39ac7c206f2e3f9e6b52\": rpc error: code = NotFound desc = could not find container \"68056e05d47196d82ebb2ffd95833b4ddf2f40239def39ac7c206f2e3f9e6b52\": container with ID starting with 68056e05d47196d82ebb2ffd95833b4ddf2f40239def39ac7c206f2e3f9e6b52 not found: ID does not exist"
Sep 09 10:40:09.916511 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:09.916488    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jtgsc"]
Sep 09 10:40:09.918304 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:09.918285    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jtgsc"]
Sep 09 10:40:10.839104 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:10.839060    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e" path="/var/lib/kubelet/pods/b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e/volumes"
Sep 09 10:40:15.875035 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:15.875005    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-shsmf"
Sep 09 10:40:18.159453 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:18.159418    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shsmf"]
Sep 09 10:40:18.159860 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:18.159627    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-shsmf" podUID="80572c9b-c043-4777-95e9-19dd97ac3574" containerName="limitador" containerID="cri-o://ce5620a2996f609ef246d55bccd3e39af384de2a1884bd6103f60dfbf15e7613" gracePeriod=30
Sep 09 10:40:18.934200 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:18.934164    2570 generic.go:356] "Generic (PLEG): container finished" podID="80572c9b-c043-4777-95e9-19dd97ac3574" containerID="ce5620a2996f609ef246d55bccd3e39af384de2a1884bd6103f60dfbf15e7613" exitCode=0
Sep 09 10:40:18.934367 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:18.934237    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-shsmf" event={"ID":"80572c9b-c043-4777-95e9-19dd97ac3574","Type":"ContainerDied","Data":"ce5620a2996f609ef246d55bccd3e39af384de2a1884bd6103f60dfbf15e7613"}
Sep 09 10:40:19.107110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:19.107085    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-shsmf"
Sep 09 10:40:19.236245 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:19.236168    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80572c9b-c043-4777-95e9-19dd97ac3574-config-file\") pod \"80572c9b-c043-4777-95e9-19dd97ac3574\" (UID: \"80572c9b-c043-4777-95e9-19dd97ac3574\") "
Sep 09 10:40:19.236245 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:19.236219    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swdgf\" (UniqueName: \"kubernetes.io/projected/80572c9b-c043-4777-95e9-19dd97ac3574-kube-api-access-swdgf\") pod \"80572c9b-c043-4777-95e9-19dd97ac3574\" (UID: \"80572c9b-c043-4777-95e9-19dd97ac3574\") "
Sep 09 10:40:19.236630 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:19.236497    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80572c9b-c043-4777-95e9-19dd97ac3574-config-file" (OuterVolumeSpecName: "config-file") pod "80572c9b-c043-4777-95e9-19dd97ac3574" (UID: "80572c9b-c043-4777-95e9-19dd97ac3574"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:40:19.238244 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:19.238222    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80572c9b-c043-4777-95e9-19dd97ac3574-kube-api-access-swdgf" (OuterVolumeSpecName: "kube-api-access-swdgf") pod "80572c9b-c043-4777-95e9-19dd97ac3574" (UID: "80572c9b-c043-4777-95e9-19dd97ac3574"). InnerVolumeSpecName "kube-api-access-swdgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:40:19.337664 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:19.337633    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swdgf\" (UniqueName: \"kubernetes.io/projected/80572c9b-c043-4777-95e9-19dd97ac3574-kube-api-access-swdgf\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:40:19.337664 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:19.337664    2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80572c9b-c043-4777-95e9-19dd97ac3574-config-file\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:40:19.939593 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:19.939560    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-shsmf"
Sep 09 10:40:19.939977 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:19.939601    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-shsmf" event={"ID":"80572c9b-c043-4777-95e9-19dd97ac3574","Type":"ContainerDied","Data":"b1576c6b3d3794eecd5c4a02d5a8f53ebe49cb3805a187ac75e03aedd110f8a7"}
Sep 09 10:40:19.939977 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:19.939661    2570 scope.go:117] "RemoveContainer" containerID="ce5620a2996f609ef246d55bccd3e39af384de2a1884bd6103f60dfbf15e7613"
Sep 09 10:40:19.959553 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:19.959526    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shsmf"]
Sep 09 10:40:19.963472 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:19.963452    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shsmf"]
Sep 09 10:40:20.839117 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:20.839087    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80572c9b-c043-4777-95e9-19dd97ac3574" path="/var/lib/kubelet/pods/80572c9b-c043-4777-95e9-19dd97ac3574/volumes"
Sep 09 10:40:24.306753 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.306698    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:24.406572 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.406532    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:24.736134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.736068    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8zz26"]
Sep 09 10:40:24.736529 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.736511    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80572c9b-c043-4777-95e9-19dd97ac3574" containerName="limitador"
Sep 09 10:40:24.736578 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.736535    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="80572c9b-c043-4777-95e9-19dd97ac3574" containerName="limitador"
Sep 09 10:40:24.736578 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.736554    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e" containerName="authorino"
Sep 09 10:40:24.736578 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.736561    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e" containerName="authorino"
Sep 09 10:40:24.736666 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.736630    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b15c66d6-02b9-4bbc-b76d-3e9b0cf92f2e" containerName="authorino"
Sep 09 10:40:24.736666 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.736639    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="80572c9b-c043-4777-95e9-19dd97ac3574" containerName="limitador"
Sep 09 10:40:24.744609 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.744588    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8zz26"
Sep 09 10:40:24.748805 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.748675    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 09 10:40:24.748925 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.748818    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 10:40:24.749425 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.749406    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fcp6q\""
Sep 09 10:40:24.749514 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.749406    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 10:40:24.750097 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.750073    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8zz26"]
Sep 09 10:40:24.773030 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.773009    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:24.778861 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.778843    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2k2q\" (UniqueName: \"kubernetes.io/projected/e4139afe-da52-483e-b181-05c72f8299d4-kube-api-access-c2k2q\") pod \"postgres-69fb4594cf-8zz26\" (UID: \"e4139afe-da52-483e-b181-05c72f8299d4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8zz26"
Sep 09 10:40:24.778944 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.778927    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e4139afe-da52-483e-b181-05c72f8299d4-data\") pod \"postgres-69fb4594cf-8zz26\" (UID: \"e4139afe-da52-483e-b181-05c72f8299d4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8zz26"
Sep 09 10:40:24.879601 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.879575    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e4139afe-da52-483e-b181-05c72f8299d4-data\") pod \"postgres-69fb4594cf-8zz26\" (UID: \"e4139afe-da52-483e-b181-05c72f8299d4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8zz26"
Sep 09 10:40:24.879743 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.879635    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2k2q\" (UniqueName: \"kubernetes.io/projected/e4139afe-da52-483e-b181-05c72f8299d4-kube-api-access-c2k2q\") pod \"postgres-69fb4594cf-8zz26\" (UID: \"e4139afe-da52-483e-b181-05c72f8299d4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8zz26"
Sep 09 10:40:24.879956 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.879936    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e4139afe-da52-483e-b181-05c72f8299d4-data\") pod \"postgres-69fb4594cf-8zz26\" (UID: \"e4139afe-da52-483e-b181-05c72f8299d4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8zz26"
Sep 09 10:40:24.888304 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:24.888283    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2k2q\" (UniqueName: \"kubernetes.io/projected/e4139afe-da52-483e-b181-05c72f8299d4-kube-api-access-c2k2q\") pod \"postgres-69fb4594cf-8zz26\" (UID: \"e4139afe-da52-483e-b181-05c72f8299d4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8zz26"
Sep 09 10:40:25.055374 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:25.055348    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8zz26"
Sep 09 10:40:25.327578 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:25.327502    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8zz26"]
Sep 09 10:40:25.331099 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:40:25.331076    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4139afe_da52_483e_b181_05c72f8299d4.slice/crio-bfd17e756692bce1cdc850f98cb3d586ef83cc410d6d320a79d52d35075828b1 WatchSource:0}: Error finding container bfd17e756692bce1cdc850f98cb3d586ef83cc410d6d320a79d52d35075828b1: Status 404 returned error can't find the container with id bfd17e756692bce1cdc850f98cb3d586ef83cc410d6d320a79d52d35075828b1
Sep 09 10:40:25.966917 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:25.966882    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8zz26" event={"ID":"e4139afe-da52-483e-b181-05c72f8299d4","Type":"ContainerStarted","Data":"bfd17e756692bce1cdc850f98cb3d586ef83cc410d6d320a79d52d35075828b1"}
Sep 09 10:40:28.983495 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:28.983407    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8zz26" event={"ID":"e4139afe-da52-483e-b181-05c72f8299d4","Type":"ContainerStarted","Data":"29ac06d41cdeed23b885a9032adc7580caae7d5e1b7e88818d77a4ba86dc8b71"}
Sep 09 10:40:28.983949 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:28.983529    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8zz26"
Sep 09 10:40:29.003089 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:29.003042    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8zz26" podStartSLOduration=1.670680202 podStartE2EDuration="5.003027992s" totalImagesPullingTime="3.33234779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:40:29.001715381 +0000 UTC m=+868.665354355" watchObservedRunningTime="2026-09-09 10:40:29.003027992 +0000 UTC m=+868.666666966"
Sep 09 10:40:35.017134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.017107    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8zz26"
Sep 09 10:40:35.561793 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.561745    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-zjh89"]
Sep 09 10:40:35.565066 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.565045    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zjh89"
Sep 09 10:40:35.572515 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.572495    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-zjh89"]
Sep 09 10:40:35.617209 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.617168    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:35.668591 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.668552    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x25zp\" (UniqueName: \"kubernetes.io/projected/9a7cc86c-65f8-4fce-97bc-09e3d6e3a655-kube-api-access-x25zp\") pod \"authorino-557f876f69-zjh89\" (UID: \"9a7cc86c-65f8-4fce-97bc-09e3d6e3a655\") " pod="kuadrant-system/authorino-557f876f69-zjh89"
Sep 09 10:40:35.717480 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.717445    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:35.769146 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.769117    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x25zp\" (UniqueName: \"kubernetes.io/projected/9a7cc86c-65f8-4fce-97bc-09e3d6e3a655-kube-api-access-x25zp\") pod \"authorino-557f876f69-zjh89\" (UID: \"9a7cc86c-65f8-4fce-97bc-09e3d6e3a655\") " pod="kuadrant-system/authorino-557f876f69-zjh89"
Sep 09 10:40:35.778058 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.778037    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x25zp\" (UniqueName: \"kubernetes.io/projected/9a7cc86c-65f8-4fce-97bc-09e3d6e3a655-kube-api-access-x25zp\") pod \"authorino-557f876f69-zjh89\" (UID: \"9a7cc86c-65f8-4fce-97bc-09e3d6e3a655\") " pod="kuadrant-system/authorino-557f876f69-zjh89"
Sep 09 10:40:35.802227 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.802202    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zjh89"]
Sep 09 10:40:35.802436 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.802425    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zjh89"
Sep 09 10:40:35.896935 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.896903    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7889699bc-2l66p"]
Sep 09 10:40:35.901158 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.901138    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7889699bc-2l66p"
Sep 09 10:40:35.903668 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.903651    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 10:40:35.910338 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.910316    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7889699bc-2l66p"]
Sep 09 10:40:35.970700 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.970666    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5fed0dea-cb39-4a3a-b1f5-fabeb080806a-tls-cert\") pod \"authorino-7889699bc-2l66p\" (UID: \"5fed0dea-cb39-4a3a-b1f5-fabeb080806a\") " pod="kuadrant-system/authorino-7889699bc-2l66p"
Sep 09 10:40:35.970861 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:35.970715    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdm6r\" (UniqueName: \"kubernetes.io/projected/5fed0dea-cb39-4a3a-b1f5-fabeb080806a-kube-api-access-sdm6r\") pod \"authorino-7889699bc-2l66p\" (UID: \"5fed0dea-cb39-4a3a-b1f5-fabeb080806a\") " pod="kuadrant-system/authorino-7889699bc-2l66p"
Sep 09 10:40:36.071542 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:36.071459    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5fed0dea-cb39-4a3a-b1f5-fabeb080806a-tls-cert\") pod \"authorino-7889699bc-2l66p\" (UID: \"5fed0dea-cb39-4a3a-b1f5-fabeb080806a\") " pod="kuadrant-system/authorino-7889699bc-2l66p"
Sep 09 10:40:36.071542 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:36.071529    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdm6r\" (UniqueName: \"kubernetes.io/projected/5fed0dea-cb39-4a3a-b1f5-fabeb080806a-kube-api-access-sdm6r\") pod \"authorino-7889699bc-2l66p\" (UID: \"5fed0dea-cb39-4a3a-b1f5-fabeb080806a\") " pod="kuadrant-system/authorino-7889699bc-2l66p"
Sep 09 10:40:36.073066 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:36.073046    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zjh89"]
Sep 09 10:40:36.074096 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:36.074074    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5fed0dea-cb39-4a3a-b1f5-fabeb080806a-tls-cert\") pod \"authorino-7889699bc-2l66p\" (UID: \"5fed0dea-cb39-4a3a-b1f5-fabeb080806a\") " pod="kuadrant-system/authorino-7889699bc-2l66p"
Sep 09 10:40:36.075212 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:40:36.075188    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a7cc86c_65f8_4fce_97bc_09e3d6e3a655.slice/crio-3fae794f0103857937ae1b7e34168dac0871456f668dd9caeb9ce2c7fee2c6bd WatchSource:0}: Error finding container 3fae794f0103857937ae1b7e34168dac0871456f668dd9caeb9ce2c7fee2c6bd: Status 404 returned error can't find the container with id 3fae794f0103857937ae1b7e34168dac0871456f668dd9caeb9ce2c7fee2c6bd
Sep 09 10:40:36.080835 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:36.080817    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdm6r\" (UniqueName: \"kubernetes.io/projected/5fed0dea-cb39-4a3a-b1f5-fabeb080806a-kube-api-access-sdm6r\") pod \"authorino-7889699bc-2l66p\" (UID: \"5fed0dea-cb39-4a3a-b1f5-fabeb080806a\") " pod="kuadrant-system/authorino-7889699bc-2l66p"
Sep 09 10:40:36.210231 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:36.210203    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7889699bc-2l66p"
Sep 09 10:40:36.478895 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:36.478870    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7889699bc-2l66p"]
Sep 09 10:40:36.481236 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:40:36.481213    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fed0dea_cb39_4a3a_b1f5_fabeb080806a.slice/crio-79a1227e48561bc80a148b3de151b2445a6437b056f9c3dc3a87ef6b93bf0124 WatchSource:0}: Error finding container 79a1227e48561bc80a148b3de151b2445a6437b056f9c3dc3a87ef6b93bf0124: Status 404 returned error can't find the container with id 79a1227e48561bc80a148b3de151b2445a6437b056f9c3dc3a87ef6b93bf0124
Sep 09 10:40:37.018739 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.018708    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7889699bc-2l66p" event={"ID":"5fed0dea-cb39-4a3a-b1f5-fabeb080806a","Type":"ContainerStarted","Data":"a00691e141ff5b1688cffa5c156d6bc936d07158ec5b70621f1efdfc837a974d"}
Sep 09 10:40:37.018954 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.018749    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7889699bc-2l66p" event={"ID":"5fed0dea-cb39-4a3a-b1f5-fabeb080806a","Type":"ContainerStarted","Data":"79a1227e48561bc80a148b3de151b2445a6437b056f9c3dc3a87ef6b93bf0124"}
Sep 09 10:40:37.020271 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.020250    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-zjh89" event={"ID":"9a7cc86c-65f8-4fce-97bc-09e3d6e3a655","Type":"ContainerStarted","Data":"71e334faa898846e8bcc4a18747c417d60c44b42180be5f0d9abb36705309276"}
Sep 09 10:40:37.020378 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.020278    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-zjh89" event={"ID":"9a7cc86c-65f8-4fce-97bc-09e3d6e3a655","Type":"ContainerStarted","Data":"3fae794f0103857937ae1b7e34168dac0871456f668dd9caeb9ce2c7fee2c6bd"}
Sep 09 10:40:37.020378 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.020323    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-zjh89" podUID="9a7cc86c-65f8-4fce-97bc-09e3d6e3a655" containerName="authorino" containerID="cri-o://71e334faa898846e8bcc4a18747c417d60c44b42180be5f0d9abb36705309276" gracePeriod=30
Sep 09 10:40:37.036545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.036509    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7889699bc-2l66p" podStartSLOduration=1.673279046 podStartE2EDuration="2.036494991s" totalImagesPullingTime="363.215945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:40:37.035282073 +0000 UTC m=+876.698921047" watchObservedRunningTime="2026-09-09 10:40:37.036494991 +0000 UTC m=+876.700133961"
Sep 09 10:40:37.051202 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.051153    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-zjh89" podStartSLOduration=1.7177344030000001 podStartE2EDuration="2.051134941s" totalImagesPullingTime="333.400538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:40:37.050787552 +0000 UTC m=+876.714426524" watchObservedRunningTime="2026-09-09 10:40:37.051134941 +0000 UTC m=+876.714773917"
Sep 09 10:40:37.069338 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.069313    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k988x"]
Sep 09 10:40:37.069524 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.069503    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-k988x" podUID="5ee36ca6-a3f4-496d-a71a-fa5dabf39f17" containerName="authorino" containerID="cri-o://a23a73a577dc1eb63755f1a96f1fe3d5b7e1e73d848932e1f1c9a24ba4e665cf" gracePeriod=30
Sep 09 10:40:37.313957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.313935    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zjh89"
Sep 09 10:40:37.318621 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.318604    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-k988x"
Sep 09 10:40:37.384585 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.384556    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x25zp\" (UniqueName: \"kubernetes.io/projected/9a7cc86c-65f8-4fce-97bc-09e3d6e3a655-kube-api-access-x25zp\") pod \"9a7cc86c-65f8-4fce-97bc-09e3d6e3a655\" (UID: \"9a7cc86c-65f8-4fce-97bc-09e3d6e3a655\") "
Sep 09 10:40:37.384729 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.384607    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v46md\" (UniqueName: \"kubernetes.io/projected/5ee36ca6-a3f4-496d-a71a-fa5dabf39f17-kube-api-access-v46md\") pod \"5ee36ca6-a3f4-496d-a71a-fa5dabf39f17\" (UID: \"5ee36ca6-a3f4-496d-a71a-fa5dabf39f17\") "
Sep 09 10:40:37.386644 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.386617    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7cc86c-65f8-4fce-97bc-09e3d6e3a655-kube-api-access-x25zp" (OuterVolumeSpecName: "kube-api-access-x25zp") pod "9a7cc86c-65f8-4fce-97bc-09e3d6e3a655" (UID: "9a7cc86c-65f8-4fce-97bc-09e3d6e3a655"). InnerVolumeSpecName "kube-api-access-x25zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:40:37.386796 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.386669    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee36ca6-a3f4-496d-a71a-fa5dabf39f17-kube-api-access-v46md" (OuterVolumeSpecName: "kube-api-access-v46md") pod "5ee36ca6-a3f4-496d-a71a-fa5dabf39f17" (UID: "5ee36ca6-a3f4-496d-a71a-fa5dabf39f17"). InnerVolumeSpecName "kube-api-access-v46md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:40:37.485812 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.485726    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x25zp\" (UniqueName: \"kubernetes.io/projected/9a7cc86c-65f8-4fce-97bc-09e3d6e3a655-kube-api-access-x25zp\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:40:37.485812 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:37.485750    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v46md\" (UniqueName: \"kubernetes.io/projected/5ee36ca6-a3f4-496d-a71a-fa5dabf39f17-kube-api-access-v46md\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:40:38.026157 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.026124    2570 generic.go:356] "Generic (PLEG): container finished" podID="5ee36ca6-a3f4-496d-a71a-fa5dabf39f17" containerID="a23a73a577dc1eb63755f1a96f1fe3d5b7e1e73d848932e1f1c9a24ba4e665cf" exitCode=0
Sep 09 10:40:38.026327 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.026175    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-k988x"
Sep 09 10:40:38.026327 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.026209    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-k988x" event={"ID":"5ee36ca6-a3f4-496d-a71a-fa5dabf39f17","Type":"ContainerDied","Data":"a23a73a577dc1eb63755f1a96f1fe3d5b7e1e73d848932e1f1c9a24ba4e665cf"}
Sep 09 10:40:38.026327 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.026247    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-k988x" event={"ID":"5ee36ca6-a3f4-496d-a71a-fa5dabf39f17","Type":"ContainerDied","Data":"ff794047012ab2828240d3286ab871a6191d812809e8e50c405b1ed432409be8"}
Sep 09 10:40:38.026327 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.026269    2570 scope.go:117] "RemoveContainer" containerID="a23a73a577dc1eb63755f1a96f1fe3d5b7e1e73d848932e1f1c9a24ba4e665cf"
Sep 09 10:40:38.027572 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.027442    2570 generic.go:356] "Generic (PLEG): container finished" podID="9a7cc86c-65f8-4fce-97bc-09e3d6e3a655" containerID="71e334faa898846e8bcc4a18747c417d60c44b42180be5f0d9abb36705309276" exitCode=0
Sep 09 10:40:38.027572 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.027473    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-zjh89" event={"ID":"9a7cc86c-65f8-4fce-97bc-09e3d6e3a655","Type":"ContainerDied","Data":"71e334faa898846e8bcc4a18747c417d60c44b42180be5f0d9abb36705309276"}
Sep 09 10:40:38.027572 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.027495    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zjh89"
Sep 09 10:40:38.027572 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.027508    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-zjh89" event={"ID":"9a7cc86c-65f8-4fce-97bc-09e3d6e3a655","Type":"ContainerDied","Data":"3fae794f0103857937ae1b7e34168dac0871456f668dd9caeb9ce2c7fee2c6bd"}
Sep 09 10:40:38.034231 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.034194    2570 scope.go:117] "RemoveContainer" containerID="a23a73a577dc1eb63755f1a96f1fe3d5b7e1e73d848932e1f1c9a24ba4e665cf"
Sep 09 10:40:38.034478 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:40:38.034460    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23a73a577dc1eb63755f1a96f1fe3d5b7e1e73d848932e1f1c9a24ba4e665cf\": container with ID starting with a23a73a577dc1eb63755f1a96f1fe3d5b7e1e73d848932e1f1c9a24ba4e665cf not found: ID does not exist" containerID="a23a73a577dc1eb63755f1a96f1fe3d5b7e1e73d848932e1f1c9a24ba4e665cf"
Sep 09 10:40:38.034543 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.034486    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23a73a577dc1eb63755f1a96f1fe3d5b7e1e73d848932e1f1c9a24ba4e665cf"} err="failed to get container status \"a23a73a577dc1eb63755f1a96f1fe3d5b7e1e73d848932e1f1c9a24ba4e665cf\": rpc error: code = NotFound desc = could not find container \"a23a73a577dc1eb63755f1a96f1fe3d5b7e1e73d848932e1f1c9a24ba4e665cf\": container with ID starting with a23a73a577dc1eb63755f1a96f1fe3d5b7e1e73d848932e1f1c9a24ba4e665cf not found: ID does not exist"
Sep 09 10:40:38.034543 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.034502    2570 scope.go:117] "RemoveContainer" containerID="71e334faa898846e8bcc4a18747c417d60c44b42180be5f0d9abb36705309276"
Sep 09 10:40:38.042075 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.042055    2570 scope.go:117] "RemoveContainer" containerID="71e334faa898846e8bcc4a18747c417d60c44b42180be5f0d9abb36705309276"
Sep 09 10:40:38.042328 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:40:38.042310    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e334faa898846e8bcc4a18747c417d60c44b42180be5f0d9abb36705309276\": container with ID starting with 71e334faa898846e8bcc4a18747c417d60c44b42180be5f0d9abb36705309276 not found: ID does not exist" containerID="71e334faa898846e8bcc4a18747c417d60c44b42180be5f0d9abb36705309276"
Sep 09 10:40:38.042397 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.042339    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e334faa898846e8bcc4a18747c417d60c44b42180be5f0d9abb36705309276"} err="failed to get container status \"71e334faa898846e8bcc4a18747c417d60c44b42180be5f0d9abb36705309276\": rpc error: code = NotFound desc = could not find container \"71e334faa898846e8bcc4a18747c417d60c44b42180be5f0d9abb36705309276\": container with ID starting with 71e334faa898846e8bcc4a18747c417d60c44b42180be5f0d9abb36705309276 not found: ID does not exist"
Sep 09 10:40:38.051463 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.051444    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k988x"]
Sep 09 10:40:38.053733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.053714    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k988x"]
Sep 09 10:40:38.064499 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.064480    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zjh89"]
Sep 09 10:40:38.073444 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.069108    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-zjh89"]
Sep 09 10:40:38.838836 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.838805    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ee36ca6-a3f4-496d-a71a-fa5dabf39f17" path="/var/lib/kubelet/pods/5ee36ca6-a3f4-496d-a71a-fa5dabf39f17/volumes"
Sep 09 10:40:38.839201 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:38.839105    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a7cc86c-65f8-4fce-97bc-09e3d6e3a655" path="/var/lib/kubelet/pods/9a7cc86c-65f8-4fce-97bc-09e3d6e3a655/volumes"
Sep 09 10:40:42.040525 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.040491    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:42.100286 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.100254    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57db658748-5vjrr"]
Sep 09 10:40:42.100591 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.100579    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ee36ca6-a3f4-496d-a71a-fa5dabf39f17" containerName="authorino"
Sep 09 10:40:42.100635 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.100593    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ee36ca6-a3f4-496d-a71a-fa5dabf39f17" containerName="authorino"
Sep 09 10:40:42.100635 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.100602    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a7cc86c-65f8-4fce-97bc-09e3d6e3a655" containerName="authorino"
Sep 09 10:40:42.100635 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.100607    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a7cc86c-65f8-4fce-97bc-09e3d6e3a655" containerName="authorino"
Sep 09 10:40:42.100729 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.100662    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ee36ca6-a3f4-496d-a71a-fa5dabf39f17" containerName="authorino"
Sep 09 10:40:42.100729 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.100675    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a7cc86c-65f8-4fce-97bc-09e3d6e3a655" containerName="authorino"
Sep 09 10:40:42.105816 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.105801    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:42.110227 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.110195    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 10:40:42.110349 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.110251    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 10:40:42.111096 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.111078    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lr9tv\""
Sep 09 10:40:42.111161 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.111127    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 10:40:42.119585 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.119561    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57db658748-5vjrr"]
Sep 09 10:40:42.146585 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.146559    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:42.222593 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.222555    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgpq\" (UniqueName: \"kubernetes.io/projected/94b7c946-7cf8-4f38-8463-d0044556d4ac-kube-api-access-9tgpq\") pod \"maas-controller-57db658748-5vjrr\" (UID: \"94b7c946-7cf8-4f38-8463-d0044556d4ac\") " pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:42.222813 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.222621    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-metrics-certs\") pod \"maas-controller-57db658748-5vjrr\" (UID: \"94b7c946-7cf8-4f38-8463-d0044556d4ac\") " pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:42.222813 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.222651    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-webhook-cert\") pod \"maas-controller-57db658748-5vjrr\" (UID: \"94b7c946-7cf8-4f38-8463-d0044556d4ac\") " pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:42.244236 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.244206    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:42.311419 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.311337    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-ln86k"]
Sep 09 10:40:42.314889 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.314868    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 10:40:42.323702 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.323678    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgpq\" (UniqueName: \"kubernetes.io/projected/94b7c946-7cf8-4f38-8463-d0044556d4ac-kube-api-access-9tgpq\") pod \"maas-controller-57db658748-5vjrr\" (UID: \"94b7c946-7cf8-4f38-8463-d0044556d4ac\") " pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:42.323822 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.323728    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-metrics-certs\") pod \"maas-controller-57db658748-5vjrr\" (UID: \"94b7c946-7cf8-4f38-8463-d0044556d4ac\") " pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:42.323822 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.323750    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-webhook-cert\") pod \"maas-controller-57db658748-5vjrr\" (UID: \"94b7c946-7cf8-4f38-8463-d0044556d4ac\") " pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:42.323983 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:40:42.323856    2570 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 09 10:40:42.323983 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:40:42.323904    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-webhook-cert podName:94b7c946-7cf8-4f38-8463-d0044556d4ac nodeName:}" failed. No retries permitted until 2026-09-09 10:40:42.823889361 +0000 UTC m=+882.487528313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-webhook-cert") pod "maas-controller-57db658748-5vjrr" (UID: "94b7c946-7cf8-4f38-8463-d0044556d4ac") : secret "maas-controller-webhook-cert" not found
Sep 09 10:40:42.326098 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.326074    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-metrics-certs\") pod \"maas-controller-57db658748-5vjrr\" (UID: \"94b7c946-7cf8-4f38-8463-d0044556d4ac\") " pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:42.330480 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.330458    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-ln86k"]
Sep 09 10:40:42.337142 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.337122    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgpq\" (UniqueName: \"kubernetes.io/projected/94b7c946-7cf8-4f38-8463-d0044556d4ac-kube-api-access-9tgpq\") pod \"maas-controller-57db658748-5vjrr\" (UID: \"94b7c946-7cf8-4f38-8463-d0044556d4ac\") " pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:42.424267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.424237    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c52d5c22-b63c-4998-970e-70d31838fd5e-webhook-cert\") pod \"maas-controller-68dd87c4cd-ln86k\" (UID: \"c52d5c22-b63c-4998-970e-70d31838fd5e\") " pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 10:40:42.424267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.424273    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72xx2\" (UniqueName: \"kubernetes.io/projected/c52d5c22-b63c-4998-970e-70d31838fd5e-kube-api-access-72xx2\") pod \"maas-controller-68dd87c4cd-ln86k\" (UID: \"c52d5c22-b63c-4998-970e-70d31838fd5e\") " pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 10:40:42.424483 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.424288    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c52d5c22-b63c-4998-970e-70d31838fd5e-metrics-certs\") pod \"maas-controller-68dd87c4cd-ln86k\" (UID: \"c52d5c22-b63c-4998-970e-70d31838fd5e\") " pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 10:40:42.525058 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.525014    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c52d5c22-b63c-4998-970e-70d31838fd5e-webhook-cert\") pod \"maas-controller-68dd87c4cd-ln86k\" (UID: \"c52d5c22-b63c-4998-970e-70d31838fd5e\") " pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 10:40:42.525058 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.525063    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72xx2\" (UniqueName: \"kubernetes.io/projected/c52d5c22-b63c-4998-970e-70d31838fd5e-kube-api-access-72xx2\") pod \"maas-controller-68dd87c4cd-ln86k\" (UID: \"c52d5c22-b63c-4998-970e-70d31838fd5e\") " pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 10:40:42.525259 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.525079    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c52d5c22-b63c-4998-970e-70d31838fd5e-metrics-certs\") pod \"maas-controller-68dd87c4cd-ln86k\" (UID: \"c52d5c22-b63c-4998-970e-70d31838fd5e\") " pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 10:40:42.527438 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.527411    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c52d5c22-b63c-4998-970e-70d31838fd5e-metrics-certs\") pod \"maas-controller-68dd87c4cd-ln86k\" (UID: \"c52d5c22-b63c-4998-970e-70d31838fd5e\") " pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 10:40:42.527545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.527441    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c52d5c22-b63c-4998-970e-70d31838fd5e-webhook-cert\") pod \"maas-controller-68dd87c4cd-ln86k\" (UID: \"c52d5c22-b63c-4998-970e-70d31838fd5e\") " pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 10:40:42.534915 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.534896    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72xx2\" (UniqueName: \"kubernetes.io/projected/c52d5c22-b63c-4998-970e-70d31838fd5e-kube-api-access-72xx2\") pod \"maas-controller-68dd87c4cd-ln86k\" (UID: \"c52d5c22-b63c-4998-970e-70d31838fd5e\") " pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 10:40:42.624515 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.624432    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 10:40:42.827507 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.827461    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-webhook-cert\") pod \"maas-controller-57db658748-5vjrr\" (UID: \"94b7c946-7cf8-4f38-8463-d0044556d4ac\") " pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:42.829828 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.829802    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-webhook-cert\") pod \"maas-controller-57db658748-5vjrr\" (UID: \"94b7c946-7cf8-4f38-8463-d0044556d4ac\") " pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:42.900394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:42.900309    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-ln86k"]
Sep 09 10:40:42.903882 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:40:42.903861    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52d5c22_b63c_4998_970e_70d31838fd5e.slice/crio-8954ab842308007e3b7c123dcc8c97ad81b8603b87130995b367bcd3cd0abf67 WatchSource:0}: Error finding container 8954ab842308007e3b7c123dcc8c97ad81b8603b87130995b367bcd3cd0abf67: Status 404 returned error can't find the container with id 8954ab842308007e3b7c123dcc8c97ad81b8603b87130995b367bcd3cd0abf67
Sep 09 10:40:43.018054 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:43.018024    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:43.052011 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:43.051980    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68dd87c4cd-ln86k" event={"ID":"c52d5c22-b63c-4998-970e-70d31838fd5e","Type":"ContainerStarted","Data":"8954ab842308007e3b7c123dcc8c97ad81b8603b87130995b367bcd3cd0abf67"}
Sep 09 10:40:43.289544 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:43.289518    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57db658748-5vjrr"]
Sep 09 10:40:43.290995 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:40:43.290969    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b7c946_7cf8_4f38_8463_d0044556d4ac.slice/crio-3738146f1d82f03eac396fdad305bb4e69005c22c7509c704339d8fd62ac5cf6 WatchSource:0}: Error finding container 3738146f1d82f03eac396fdad305bb4e69005c22c7509c704339d8fd62ac5cf6: Status 404 returned error can't find the container with id 3738146f1d82f03eac396fdad305bb4e69005c22c7509c704339d8fd62ac5cf6
Sep 09 10:40:43.387295 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:43.387261    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:43.479989 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:43.479959    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:44.061822 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:44.061786    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57db658748-5vjrr" event={"ID":"94b7c946-7cf8-4f38-8463-d0044556d4ac","Type":"ContainerStarted","Data":"3738146f1d82f03eac396fdad305bb4e69005c22c7509c704339d8fd62ac5cf6"}
Sep 09 10:40:45.965281 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:45.965235    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:46.072157 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:46.072071    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:40:46.073972 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:46.073946    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57db658748-5vjrr" event={"ID":"94b7c946-7cf8-4f38-8463-d0044556d4ac","Type":"ContainerStarted","Data":"6bf51494e19915a1dd60cdab01a408ac0430ba71426302a8bd2c5bfa880b6b3c"}
Sep 09 10:40:46.074113 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:46.074012    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:46.075417 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:46.075395    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68dd87c4cd-ln86k" event={"ID":"c52d5c22-b63c-4998-970e-70d31838fd5e","Type":"ContainerStarted","Data":"4455b0356978e769ec98de5ac0cce98c394ec7fed7a5f92dbadbd1ddb8226b5e"}
Sep 09 10:40:46.075511 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:46.075439    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 10:40:46.143553 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:46.143509    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57db658748-5vjrr" podStartSLOduration=1.631741425 podStartE2EDuration="4.143494525s" totalImagesPullingTime="2.5117531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:40:46.141954724 +0000 UTC m=+885.805593698" watchObservedRunningTime="2026-09-09 10:40:46.143494525 +0000 UTC m=+885.807133499"
Sep 09 10:40:57.082688 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.082657    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 10:40:57.083149 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.082722    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:57.103640 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.103597    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68dd87c4cd-ln86k" podStartSLOduration=12.214483048 podStartE2EDuration="15.10358355s" totalImagesPullingTime="2.889100502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:40:46.188087259 +0000 UTC m=+885.851726232" watchObservedRunningTime="2026-09-09 10:40:57.10358355 +0000 UTC m=+896.767222533"
Sep 09 10:40:57.154711 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.154678    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57db658748-5vjrr"]
Sep 09 10:40:57.155173 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.155138    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57db658748-5vjrr" podUID="94b7c946-7cf8-4f38-8463-d0044556d4ac" containerName="manager" containerID="cri-o://6bf51494e19915a1dd60cdab01a408ac0430ba71426302a8bd2c5bfa880b6b3c" gracePeriod=30
Sep 09 10:40:57.391633 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.391612    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:57.554162 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.554126    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tgpq\" (UniqueName: \"kubernetes.io/projected/94b7c946-7cf8-4f38-8463-d0044556d4ac-kube-api-access-9tgpq\") pod \"94b7c946-7cf8-4f38-8463-d0044556d4ac\" (UID: \"94b7c946-7cf8-4f38-8463-d0044556d4ac\") "
Sep 09 10:40:57.554328 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.554210    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-webhook-cert\") pod \"94b7c946-7cf8-4f38-8463-d0044556d4ac\" (UID: \"94b7c946-7cf8-4f38-8463-d0044556d4ac\") "
Sep 09 10:40:57.554328 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.554256    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-metrics-certs\") pod \"94b7c946-7cf8-4f38-8463-d0044556d4ac\" (UID: \"94b7c946-7cf8-4f38-8463-d0044556d4ac\") "
Sep 09 10:40:57.556316 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.556286    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "94b7c946-7cf8-4f38-8463-d0044556d4ac" (UID: "94b7c946-7cf8-4f38-8463-d0044556d4ac"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:40:57.556316 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.556303    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "94b7c946-7cf8-4f38-8463-d0044556d4ac" (UID: "94b7c946-7cf8-4f38-8463-d0044556d4ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:40:57.556471 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.556366    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b7c946-7cf8-4f38-8463-d0044556d4ac-kube-api-access-9tgpq" (OuterVolumeSpecName: "kube-api-access-9tgpq") pod "94b7c946-7cf8-4f38-8463-d0044556d4ac" (UID: "94b7c946-7cf8-4f38-8463-d0044556d4ac"). InnerVolumeSpecName "kube-api-access-9tgpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:40:57.655041 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.654978    2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-webhook-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:40:57.655041 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.654999    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94b7c946-7cf8-4f38-8463-d0044556d4ac-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:40:57.655041 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:57.655008    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tgpq\" (UniqueName: \"kubernetes.io/projected/94b7c946-7cf8-4f38-8463-d0044556d4ac-kube-api-access-9tgpq\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:40:58.131953 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:58.129540    2570 generic.go:356] "Generic (PLEG): container finished" podID="94b7c946-7cf8-4f38-8463-d0044556d4ac" containerID="6bf51494e19915a1dd60cdab01a408ac0430ba71426302a8bd2c5bfa880b6b3c" exitCode=0
Sep 09 10:40:58.131953 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:58.129668    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57db658748-5vjrr" event={"ID":"94b7c946-7cf8-4f38-8463-d0044556d4ac","Type":"ContainerDied","Data":"6bf51494e19915a1dd60cdab01a408ac0430ba71426302a8bd2c5bfa880b6b3c"}
Sep 09 10:40:58.131953 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:58.129714    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57db658748-5vjrr" event={"ID":"94b7c946-7cf8-4f38-8463-d0044556d4ac","Type":"ContainerDied","Data":"3738146f1d82f03eac396fdad305bb4e69005c22c7509c704339d8fd62ac5cf6"}
Sep 09 10:40:58.131953 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:58.129741    2570 scope.go:117] "RemoveContainer" containerID="6bf51494e19915a1dd60cdab01a408ac0430ba71426302a8bd2c5bfa880b6b3c"
Sep 09 10:40:58.131953 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:58.129922    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57db658748-5vjrr"
Sep 09 10:40:58.141351 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:58.141330    2570 scope.go:117] "RemoveContainer" containerID="6bf51494e19915a1dd60cdab01a408ac0430ba71426302a8bd2c5bfa880b6b3c"
Sep 09 10:40:58.141603 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:40:58.141585    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf51494e19915a1dd60cdab01a408ac0430ba71426302a8bd2c5bfa880b6b3c\": container with ID starting with 6bf51494e19915a1dd60cdab01a408ac0430ba71426302a8bd2c5bfa880b6b3c not found: ID does not exist" containerID="6bf51494e19915a1dd60cdab01a408ac0430ba71426302a8bd2c5bfa880b6b3c"
Sep 09 10:40:58.141649 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:58.141612    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf51494e19915a1dd60cdab01a408ac0430ba71426302a8bd2c5bfa880b6b3c"} err="failed to get container status \"6bf51494e19915a1dd60cdab01a408ac0430ba71426302a8bd2c5bfa880b6b3c\": rpc error: code = NotFound desc = could not find container \"6bf51494e19915a1dd60cdab01a408ac0430ba71426302a8bd2c5bfa880b6b3c\": container with ID starting with 6bf51494e19915a1dd60cdab01a408ac0430ba71426302a8bd2c5bfa880b6b3c not found: ID does not exist"
Sep 09 10:40:58.155224 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:58.155199    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57db658748-5vjrr"]
Sep 09 10:40:58.163484 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:58.163463    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57db658748-5vjrr"]
Sep 09 10:40:58.840443 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:40:58.840415    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94b7c946-7cf8-4f38-8463-d0044556d4ac" path="/var/lib/kubelet/pods/94b7c946-7cf8-4f38-8463-d0044556d4ac/volumes"
Sep 09 10:41:01.023981 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:01.023886    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:41:01.036247 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:01.026319    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:41:01.036247 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:01.030066    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:41:01.036247 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:01.032616    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:41:17.674617 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:17.674585    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:41:19.289694 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.289653    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7998845878-vrtrh"]
Sep 09 10:41:19.290203 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.290182    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b7c946-7cf8-4f38-8463-d0044556d4ac" containerName="manager"
Sep 09 10:41:19.290274 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.290207    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b7c946-7cf8-4f38-8463-d0044556d4ac" containerName="manager"
Sep 09 10:41:19.290326 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.290287    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b7c946-7cf8-4f38-8463-d0044556d4ac" containerName="manager"
Sep 09 10:41:19.293605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.293584    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.296754 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.296733    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 10:41:19.297051 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.297032    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 10:41:19.297051 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.297047    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k8bm4\""
Sep 09 10:41:19.304827 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.304805    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7998845878-vrtrh"]
Sep 09 10:41:19.326422 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.326388    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-maas-api-tls\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.326592 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.326447    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6ws7\" (UniqueName: \"kubernetes.io/projected/5752135e-f5b1-4ef7-a5bb-29598d19e464-kube-api-access-s6ws7\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.326592 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.326481    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5752135e-f5b1-4ef7-a5bb-29598d19e464-openshift-service-ca\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.326592 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.326505    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-metrics-certs\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.427851 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.427813    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-maas-api-tls\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.428014 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.427869    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6ws7\" (UniqueName: \"kubernetes.io/projected/5752135e-f5b1-4ef7-a5bb-29598d19e464-kube-api-access-s6ws7\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.428014 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.427898    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5752135e-f5b1-4ef7-a5bb-29598d19e464-openshift-service-ca\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.428014 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.427916    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-metrics-certs\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.428014 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:41:19.427960    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 09 10:41:19.428181 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:41:19.428028    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 09 10:41:19.428181 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:41:19.428038    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-maas-api-tls podName:5752135e-f5b1-4ef7-a5bb-29598d19e464 nodeName:}" failed. No retries permitted until 2026-09-09 10:41:19.928016379 +0000 UTC m=+919.591655331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-maas-api-tls") pod "maas-api-7998845878-vrtrh" (UID: "5752135e-f5b1-4ef7-a5bb-29598d19e464") : secret "maas-api-serving-cert" not found
Sep 09 10:41:19.428181 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:41:19.428093    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-metrics-certs podName:5752135e-f5b1-4ef7-a5bb-29598d19e464 nodeName:}" failed. No retries permitted until 2026-09-09 10:41:19.928074535 +0000 UTC m=+919.591713509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-metrics-certs") pod "maas-api-7998845878-vrtrh" (UID: "5752135e-f5b1-4ef7-a5bb-29598d19e464") : secret "maas-api-metrics-tls" not found
Sep 09 10:41:19.428575 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.428558    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5752135e-f5b1-4ef7-a5bb-29598d19e464-openshift-service-ca\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.439780 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.439742    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6ws7\" (UniqueName: \"kubernetes.io/projected/5752135e-f5b1-4ef7-a5bb-29598d19e464-kube-api-access-s6ws7\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.863222 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.863194    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb"]
Sep 09 10:41:19.868744 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.868723    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb"
Sep 09 10:41:19.872866 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.872842    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 10:41:19.872979 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.872854    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lspjv\""
Sep 09 10:41:19.876385 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.876359    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb"]
Sep 09 10:41:19.932669 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.932544    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb7e67c-33c8-4efd-9b54-41eae6add320-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-jqmwb\" (UID: \"0fb7e67c-33c8-4efd-9b54-41eae6add320\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb"
Sep 09 10:41:19.932669 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.932606    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-maas-api-tls\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.932669 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.932631    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhgf\" (UniqueName: \"kubernetes.io/projected/0fb7e67c-33c8-4efd-9b54-41eae6add320-kube-api-access-5zhgf\") pod \"payload-pre-processing-59fbc47bf8-jqmwb\" (UID: \"0fb7e67c-33c8-4efd-9b54-41eae6add320\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb"
Sep 09 10:41:19.932997 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.932779    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-metrics-certs\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.935387 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.935362    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-maas-api-tls\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:19.935671 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:19.935653    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-metrics-certs\") pod \"maas-api-7998845878-vrtrh\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") " pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:20.034032 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:20.033980    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhgf\" (UniqueName: \"kubernetes.io/projected/0fb7e67c-33c8-4efd-9b54-41eae6add320-kube-api-access-5zhgf\") pod \"payload-pre-processing-59fbc47bf8-jqmwb\" (UID: \"0fb7e67c-33c8-4efd-9b54-41eae6add320\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb"
Sep 09 10:41:20.034221 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:20.034112    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb7e67c-33c8-4efd-9b54-41eae6add320-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-jqmwb\" (UID: \"0fb7e67c-33c8-4efd-9b54-41eae6add320\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb"
Sep 09 10:41:20.034843 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:20.034816    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb7e67c-33c8-4efd-9b54-41eae6add320-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-jqmwb\" (UID: \"0fb7e67c-33c8-4efd-9b54-41eae6add320\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb"
Sep 09 10:41:20.049556 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:20.049513    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhgf\" (UniqueName: \"kubernetes.io/projected/0fb7e67c-33c8-4efd-9b54-41eae6add320-kube-api-access-5zhgf\") pod \"payload-pre-processing-59fbc47bf8-jqmwb\" (UID: \"0fb7e67c-33c8-4efd-9b54-41eae6add320\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb"
Sep 09 10:41:20.180505 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:20.180400    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb"
Sep 09 10:41:20.203548 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:20.203520    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:20.288244 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:20.288212    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:41:20.452689 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:20.452669    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb"]
Sep 09 10:41:20.454936 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:41:20.454912    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb7e67c_33c8_4efd_9b54_41eae6add320.slice/crio-930db95cc76c298f0081239aa71ca1538ff61e65e395fd69741187dc15e86998 WatchSource:0}: Error finding container 930db95cc76c298f0081239aa71ca1538ff61e65e395fd69741187dc15e86998: Status 404 returned error can't find the container with id 930db95cc76c298f0081239aa71ca1538ff61e65e395fd69741187dc15e86998
Sep 09 10:41:20.488328 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:20.488302    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7998845878-vrtrh"]
Sep 09 10:41:20.490230 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:41:20.490199    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5752135e_f5b1_4ef7_a5bb_29598d19e464.slice/crio-87140ee77830ba257cccac5f63d564cb8bff08905bc5aa30af5980b3b4e6bef2 WatchSource:0}: Error finding container 87140ee77830ba257cccac5f63d564cb8bff08905bc5aa30af5980b3b4e6bef2: Status 404 returned error can't find the container with id 87140ee77830ba257cccac5f63d564cb8bff08905bc5aa30af5980b3b4e6bef2
Sep 09 10:41:21.235547 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:21.235511    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb" event={"ID":"0fb7e67c-33c8-4efd-9b54-41eae6add320","Type":"ContainerStarted","Data":"930db95cc76c298f0081239aa71ca1538ff61e65e395fd69741187dc15e86998"}
Sep 09 10:41:21.236684 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:21.236656    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh" event={"ID":"5752135e-f5b1-4ef7-a5bb-29598d19e464","Type":"ContainerStarted","Data":"87140ee77830ba257cccac5f63d564cb8bff08905bc5aa30af5980b3b4e6bef2"}
Sep 09 10:41:22.971511 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:22.971415    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:41:23.073169 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:23.073043    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:41:23.268733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:23.268690    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh" event={"ID":"5752135e-f5b1-4ef7-a5bb-29598d19e464","Type":"ContainerStarted","Data":"471e7229b93ebf3e5758ebf9a620dd915e4875573ec0c176271aad7d9b96b1ea"}
Sep 09 10:41:23.268964 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:23.268929    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:23.295040 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:23.290243    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh" podStartSLOduration=2.104693773 podStartE2EDuration="4.290224347s" totalImagesPullingTime="2.185530574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:41:23.287678358 +0000 UTC m=+922.951317352" watchObservedRunningTime="2026-09-09 10:41:23.290224347 +0000 UTC m=+922.953863324"
Sep 09 10:41:25.279092 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:25.279057    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb" event={"ID":"0fb7e67c-33c8-4efd-9b54-41eae6add320","Type":"ContainerStarted","Data":"27f3f73aff745d228076af58413f2620d55c1bb3138d3fc0696c823d14031c22"}
Sep 09 10:41:25.279494 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:25.279236    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb"
Sep 09 10:41:25.300842 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:25.300800    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb" podStartSLOduration=2.550856031 podStartE2EDuration="6.300786476s" totalImagesPullingTime="3.749930445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:41:25.299566419 +0000 UTC m=+924.963205395" watchObservedRunningTime="2026-09-09 10:41:25.300786476 +0000 UTC m=+924.964425446"
Sep 09 10:41:29.279164 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:29.279142    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 10:41:30.247292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:30.247258    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:41:30.302558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:30.302529    2570 generic.go:356] "Generic (PLEG): container finished" podID="cd184f91-b4c9-4291-9056-e6529b1f8541" containerID="d63707c37670f4573172911380fa70e619fb68a14b143fca7d614fffdf2a08e1" exitCode=0
Sep 09 10:41:30.302954 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:30.302572    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z" event={"ID":"cd184f91-b4c9-4291-9056-e6529b1f8541","Type":"ContainerDied","Data":"d63707c37670f4573172911380fa70e619fb68a14b143fca7d614fffdf2a08e1"}
Sep 09 10:41:30.302954 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:30.302943    2570 scope.go:117] "RemoveContainer" containerID="d63707c37670f4573172911380fa70e619fb68a14b143fca7d614fffdf2a08e1"
Sep 09 10:41:30.303971 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:30.303938    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:41:30.304046 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:30.304013    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:41:30.304086 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:30.304046    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:41:30.346243 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:30.346208    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:41:31.308601 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:31.308563    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z" event={"ID":"cd184f91-b4c9-4291-9056-e6529b1f8541","Type":"ContainerStarted","Data":"2edf926d8c24e79e45d78f26c0c933125ac8dd6756a56de569b1e59176c7749a"}
Sep 09 10:41:31.387838 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:31.387805    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:41:31.388074 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:31.388032    2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lv97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 09 10:41:31.388135 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:31.388116    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z" podUID="cd184f91-b4c9-4291-9056-e6529b1f8541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 09 10:41:32.388148 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:32.388109    2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lv97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 09 10:41:32.388492 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:32.388163    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z" podUID="cd184f91-b4c9-4291-9056-e6529b1f8541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 09 10:41:33.388315 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:33.388277    2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lv97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 09 10:41:33.388677 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:33.388334    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z" podUID="cd184f91-b4c9-4291-9056-e6529b1f8541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 09 10:41:34.388125 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:34.388090    2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lv97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 09 10:41:34.388300 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:34.388152    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z" podUID="cd184f91-b4c9-4291-9056-e6529b1f8541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 09 10:41:35.387933 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:35.387895    2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lv97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 09 10:41:35.388390 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:35.387957    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z" podUID="cd184f91-b4c9-4291-9056-e6529b1f8541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 09 10:41:36.285568 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:36.285536    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jqmwb"
Sep 09 10:41:36.391037 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:36.391010    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:41:36.391404 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:36.391276    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:41:36.392088 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:36.392071    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lv97z"
Sep 09 10:41:45.011213 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.011178    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"]
Sep 09 10:41:45.015688 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.015667    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.019999 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.019978    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 10:41:45.020139 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.020060    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 10:41:45.020139 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.020070    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2mjpr\""
Sep 09 10:41:45.020139 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.020060    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 09 10:41:45.022522 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.022500    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"]
Sep 09 10:41:45.035940 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.035914    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:41:45.076551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.076519    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/846ad5fa-5f7f-4478-84c2-51884387ac91-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.076680 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.076557    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/846ad5fa-5f7f-4478-84c2-51884387ac91-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.076680 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.076653    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/846ad5fa-5f7f-4478-84c2-51884387ac91-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.076783 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.076686    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846ad5fa-5f7f-4478-84c2-51884387ac91-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.076783 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.076708    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/846ad5fa-5f7f-4478-84c2-51884387ac91-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.076783 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.076754    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49w4\" (UniqueName: \"kubernetes.io/projected/846ad5fa-5f7f-4478-84c2-51884387ac91-kube-api-access-d49w4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.177574 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.177545    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/846ad5fa-5f7f-4478-84c2-51884387ac91-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.177724 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.177579    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49w4\" (UniqueName: \"kubernetes.io/projected/846ad5fa-5f7f-4478-84c2-51884387ac91-kube-api-access-d49w4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.177724 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.177629    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/846ad5fa-5f7f-4478-84c2-51884387ac91-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.177724 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.177655    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/846ad5fa-5f7f-4478-84c2-51884387ac91-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.177724 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.177718    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/846ad5fa-5f7f-4478-84c2-51884387ac91-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.177936 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.177755    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846ad5fa-5f7f-4478-84c2-51884387ac91-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.178181 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.178148    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/846ad5fa-5f7f-4478-84c2-51884387ac91-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.178294 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.178259    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/846ad5fa-5f7f-4478-84c2-51884387ac91-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.178294 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.178284    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846ad5fa-5f7f-4478-84c2-51884387ac91-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.180078 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.180057    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/846ad5fa-5f7f-4478-84c2-51884387ac91-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.180303 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.180287    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/846ad5fa-5f7f-4478-84c2-51884387ac91-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.186540 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.186520    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49w4\" (UniqueName: \"kubernetes.io/projected/846ad5fa-5f7f-4478-84c2-51884387ac91-kube-api-access-d49w4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b\" (UID: \"846ad5fa-5f7f-4478-84c2-51884387ac91\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.327205 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.327113    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:45.605337 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:45.605306    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"]
Sep 09 10:41:45.607055 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:41:45.607008    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846ad5fa_5f7f_4478_84c2_51884387ac91.slice/crio-740f56d5587e56cbf7680b3e4a962ac275dbd0db72522bf2eeb9b7e19483a9d7 WatchSource:0}: Error finding container 740f56d5587e56cbf7680b3e4a962ac275dbd0db72522bf2eeb9b7e19483a9d7: Status 404 returned error can't find the container with id 740f56d5587e56cbf7680b3e4a962ac275dbd0db72522bf2eeb9b7e19483a9d7
Sep 09 10:41:46.385127 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:46.385100    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b" event={"ID":"846ad5fa-5f7f-4478-84c2-51884387ac91","Type":"ContainerStarted","Data":"740f56d5587e56cbf7680b3e4a962ac275dbd0db72522bf2eeb9b7e19483a9d7"}
Sep 09 10:41:47.391365 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:47.391334    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b" event={"ID":"846ad5fa-5f7f-4478-84c2-51884387ac91","Type":"ContainerStarted","Data":"09b65eefd37e4f9b917d8838179fb41862d2440cbdf1af9a79a026b62ec273ca"}
Sep 09 10:41:47.411600 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:47.411550    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b" podStartSLOduration=2.290117902 podStartE2EDuration="3.411534245s" totalImagesPullingTime="1.121416343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:41:47.409246059 +0000 UTC m=+947.072885030" watchObservedRunningTime="2026-09-09 10:41:47.411534245 +0000 UTC m=+947.075173219"
Sep 09 10:41:55.328066 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:55.328014    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:55.328453 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:55.328095    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:55.340676 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:55.340650    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:41:55.439017 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:41:55.438991    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b"
Sep 09 10:42:08.240423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:08.240390    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:42:18.622141 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:18.622105    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:42:25.423151 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:25.423071    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:42:31.027616 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.027587    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"]
Sep 09 10:42:31.030996 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.030980    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.033858 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.033838    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 09 10:42:31.040950 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.040932    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"]
Sep 09 10:42:31.060522 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.060503    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:42:31.166164 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.166139    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e4ba653-6a69-45b3-9c10-b7502cbeb628-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.166290 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.166168    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e4ba653-6a69-45b3-9c10-b7502cbeb628-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.166290 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.166189    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm7d2\" (UniqueName: \"kubernetes.io/projected/0e4ba653-6a69-45b3-9c10-b7502cbeb628-kube-api-access-nm7d2\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.166290 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.166262    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e4ba653-6a69-45b3-9c10-b7502cbeb628-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.166290 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.166287    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e4ba653-6a69-45b3-9c10-b7502cbeb628-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.166431 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.166311    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e4ba653-6a69-45b3-9c10-b7502cbeb628-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.267173 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.267140    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e4ba653-6a69-45b3-9c10-b7502cbeb628-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.267310 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.267206    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e4ba653-6a69-45b3-9c10-b7502cbeb628-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.267310 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.267235    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e4ba653-6a69-45b3-9c10-b7502cbeb628-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.267310 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.267260    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm7d2\" (UniqueName: \"kubernetes.io/projected/0e4ba653-6a69-45b3-9c10-b7502cbeb628-kube-api-access-nm7d2\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.267455 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.267328    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e4ba653-6a69-45b3-9c10-b7502cbeb628-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.267455 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.267358    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e4ba653-6a69-45b3-9c10-b7502cbeb628-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.267619 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.267598    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e4ba653-6a69-45b3-9c10-b7502cbeb628-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.267690 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.267644    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e4ba653-6a69-45b3-9c10-b7502cbeb628-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.267737 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.267717    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e4ba653-6a69-45b3-9c10-b7502cbeb628-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.269649 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.269630    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e4ba653-6a69-45b3-9c10-b7502cbeb628-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.270006 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.269986    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e4ba653-6a69-45b3-9c10-b7502cbeb628-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.277010 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.276978    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm7d2\" (UniqueName: \"kubernetes.io/projected/0e4ba653-6a69-45b3-9c10-b7502cbeb628-kube-api-access-nm7d2\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26\" (UID: \"0e4ba653-6a69-45b3-9c10-b7502cbeb628\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.341733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.341679    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:31.617529 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.617495    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"]
Sep 09 10:42:31.618626 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:42:31.618595    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4ba653_6a69_45b3_9c10_b7502cbeb628.slice/crio-980a3913794d357d124970332a5385832ccfc29e9886d106cb54ee246660bdce WatchSource:0}: Error finding container 980a3913794d357d124970332a5385832ccfc29e9886d106cb54ee246660bdce: Status 404 returned error can't find the container with id 980a3913794d357d124970332a5385832ccfc29e9886d106cb54ee246660bdce
Sep 09 10:42:31.619884 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:31.619867    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 10:42:32.601418 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:32.601384    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26" event={"ID":"0e4ba653-6a69-45b3-9c10-b7502cbeb628","Type":"ContainerStarted","Data":"8ed9408a76f6053ffc08fe5630f284af6ab5f8cc42be557517d30aad69d23c13"}
Sep 09 10:42:32.601837 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:32.601439    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26" event={"ID":"0e4ba653-6a69-45b3-9c10-b7502cbeb628","Type":"ContainerStarted","Data":"980a3913794d357d124970332a5385832ccfc29e9886d106cb54ee246660bdce"}
Sep 09 10:42:32.638876 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:32.638822    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26" podStartSLOduration=1.4361389359999999 podStartE2EDuration="1.638805862s" totalImagesPullingTime="202.666926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:42:32.638326537 +0000 UTC m=+992.301965511" watchObservedRunningTime="2026-09-09 10:42:32.638805862 +0000 UTC m=+992.302444836"
Sep 09 10:42:41.342533 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:41.342497    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:41.343013 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:41.342574    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:41.354720 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:41.354702    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:41.652498 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:41.652423    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26"
Sep 09 10:42:46.708647 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:46.708615    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:42:51.698716 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.698682    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"]
Sep 09 10:42:51.702145 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.702125    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.705673 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.705651    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 10:42:51.720187 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.720165    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"]
Sep 09 10:42:51.751799 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.751751    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:42:51.840839 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.840817    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828e34e7-50dc-4cd4-a160-e24b54870701-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.840962 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.840873    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828e34e7-50dc-4cd4-a160-e24b54870701-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.840962 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.840894    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c82xm\" (UniqueName: \"kubernetes.io/projected/828e34e7-50dc-4cd4-a160-e24b54870701-kube-api-access-c82xm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.840962 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.840923    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828e34e7-50dc-4cd4-a160-e24b54870701-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.840962 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.840938    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828e34e7-50dc-4cd4-a160-e24b54870701-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.841095 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.840972    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828e34e7-50dc-4cd4-a160-e24b54870701-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.942360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.942329    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828e34e7-50dc-4cd4-a160-e24b54870701-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.942526 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.942443    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828e34e7-50dc-4cd4-a160-e24b54870701-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.942526 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.942473    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c82xm\" (UniqueName: \"kubernetes.io/projected/828e34e7-50dc-4cd4-a160-e24b54870701-kube-api-access-c82xm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.942648 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.942543    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828e34e7-50dc-4cd4-a160-e24b54870701-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.942648 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.942576    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828e34e7-50dc-4cd4-a160-e24b54870701-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.942648 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.942623    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828e34e7-50dc-4cd4-a160-e24b54870701-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.942835 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.942752    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828e34e7-50dc-4cd4-a160-e24b54870701-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.942967 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.942944    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828e34e7-50dc-4cd4-a160-e24b54870701-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.943084 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.943054    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828e34e7-50dc-4cd4-a160-e24b54870701-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.944705 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.944683    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828e34e7-50dc-4cd4-a160-e24b54870701-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.945242 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.945226    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828e34e7-50dc-4cd4-a160-e24b54870701-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:51.951695 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:51.951636    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c82xm\" (UniqueName: \"kubernetes.io/projected/828e34e7-50dc-4cd4-a160-e24b54870701-kube-api-access-c82xm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz\" (UID: \"828e34e7-50dc-4cd4-a160-e24b54870701\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:52.013104 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:52.013082    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:42:52.284943 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:52.284915    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"]
Sep 09 10:42:52.693816 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:52.693783    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz" event={"ID":"828e34e7-50dc-4cd4-a160-e24b54870701","Type":"ContainerStarted","Data":"1bd1d0969596fe0ae732bdd8937a86f885db55dc0a0dcea6655f335030a78d58"}
Sep 09 10:42:52.693968 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:52.693832    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz" event={"ID":"828e34e7-50dc-4cd4-a160-e24b54870701","Type":"ContainerStarted","Data":"e8be7165ba2021f18f002e3523ee1e9b00d616aedafbdb981980c9496b75ad07"}
Sep 09 10:42:52.715206 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:42:52.715161    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz" podStartSLOduration=1.52490025 podStartE2EDuration="1.715147136s" totalImagesPullingTime="190.246886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:42:52.713589936 +0000 UTC m=+1012.377228904" watchObservedRunningTime="2026-09-09 10:42:52.715147136 +0000 UTC m=+1012.378786110"
Sep 09 10:43:00.510791 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.510629    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"]
Sep 09 10:43:00.515696 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.515669    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.521852 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.521825    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 10:43:00.522880 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.522847    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"]
Sep 09 10:43:00.537351 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.537327    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:43:00.615653 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.615622    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700b8066-3b32-437a-bd61-3a400c6a1229-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.615809 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.615684    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk8m9\" (UniqueName: \"kubernetes.io/projected/700b8066-3b32-437a-bd61-3a400c6a1229-kube-api-access-hk8m9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.615809 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.615718    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/700b8066-3b32-437a-bd61-3a400c6a1229-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.615809 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.615738    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/700b8066-3b32-437a-bd61-3a400c6a1229-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.615809 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.615786    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/700b8066-3b32-437a-bd61-3a400c6a1229-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.615985 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.615828    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/700b8066-3b32-437a-bd61-3a400c6a1229-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.716924 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.716896    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700b8066-3b32-437a-bd61-3a400c6a1229-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.717075 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.716949    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk8m9\" (UniqueName: \"kubernetes.io/projected/700b8066-3b32-437a-bd61-3a400c6a1229-kube-api-access-hk8m9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.717075 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.716974    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/700b8066-3b32-437a-bd61-3a400c6a1229-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.717075 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.716995    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/700b8066-3b32-437a-bd61-3a400c6a1229-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.717075 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.717014    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/700b8066-3b32-437a-bd61-3a400c6a1229-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.717075 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.717035    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/700b8066-3b32-437a-bd61-3a400c6a1229-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.717401 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.717380    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/700b8066-3b32-437a-bd61-3a400c6a1229-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.717472 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.717455    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/700b8066-3b32-437a-bd61-3a400c6a1229-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.717509 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.717474    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/700b8066-3b32-437a-bd61-3a400c6a1229-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.719180 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.719161    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/700b8066-3b32-437a-bd61-3a400c6a1229-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.722134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.722112    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 10:43:00.729288 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.729263    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk8m9\" (UniqueName: \"kubernetes.io/projected/700b8066-3b32-437a-bd61-3a400c6a1229-kube-api-access-hk8m9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.729365 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.729299    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700b8066-3b32-437a-bd61-3a400c6a1229-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4v46d\" (UID: \"700b8066-3b32-437a-bd61-3a400c6a1229\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:00.827427 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:00.827368    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:01.116890 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:01.116870    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"]
Sep 09 10:43:01.118953 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:43:01.118927    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700b8066_3b32_437a_bd61_3a400c6a1229.slice/crio-83f162a94c1c799ab4821fb0a288c03bcd07437f86078e331384999d8831b7db WatchSource:0}: Error finding container 83f162a94c1c799ab4821fb0a288c03bcd07437f86078e331384999d8831b7db: Status 404 returned error can't find the container with id 83f162a94c1c799ab4821fb0a288c03bcd07437f86078e331384999d8831b7db
Sep 09 10:43:01.735114 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:01.735033    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d" event={"ID":"700b8066-3b32-437a-bd61-3a400c6a1229","Type":"ContainerStarted","Data":"f59b72a32a8e4f126567de26bf4bdb783991432f48c064203361b4de3121a6fc"}
Sep 09 10:43:01.735114 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:01.735079    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d" event={"ID":"700b8066-3b32-437a-bd61-3a400c6a1229","Type":"ContainerStarted","Data":"83f162a94c1c799ab4821fb0a288c03bcd07437f86078e331384999d8831b7db"}
Sep 09 10:43:01.765233 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:01.765180    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d" podStartSLOduration=1.5681917680000002 podStartE2EDuration="1.765161583s" totalImagesPullingTime="196.969815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:43:01.762561274 +0000 UTC m=+1021.426200261" watchObservedRunningTime="2026-09-09 10:43:01.765161583 +0000 UTC m=+1021.428800557"
Sep 09 10:43:02.013404 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:02.013357    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:43:02.013589 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:02.013416    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:43:02.025863 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:02.025837    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:43:02.751630 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:02.751599    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz"
Sep 09 10:43:10.827821 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:10.827777    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:10.828235 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:10.827870    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:10.840419 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:10.840391    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:11.803701 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:11.803671    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4v46d"
Sep 09 10:43:14.769047 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:14.769012    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:43:29.108703 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:43:29.108667    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:13.114538 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:13.114502    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:13.216202 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:13.216171    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:20.564292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:20.564263    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7889699bc-2l66p"]
Sep 09 10:44:20.564669 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:20.564448    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7889699bc-2l66p" podUID="5fed0dea-cb39-4a3a-b1f5-fabeb080806a" containerName="authorino" containerID="cri-o://a00691e141ff5b1688cffa5c156d6bc936d07158ec5b70621f1efdfc837a974d" gracePeriod=30
Sep 09 10:44:20.806029 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:20.806007    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7889699bc-2l66p"
Sep 09 10:44:20.889064 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:20.888988    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdm6r\" (UniqueName: \"kubernetes.io/projected/5fed0dea-cb39-4a3a-b1f5-fabeb080806a-kube-api-access-sdm6r\") pod \"5fed0dea-cb39-4a3a-b1f5-fabeb080806a\" (UID: \"5fed0dea-cb39-4a3a-b1f5-fabeb080806a\") "
Sep 09 10:44:20.889064 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:20.889055    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5fed0dea-cb39-4a3a-b1f5-fabeb080806a-tls-cert\") pod \"5fed0dea-cb39-4a3a-b1f5-fabeb080806a\" (UID: \"5fed0dea-cb39-4a3a-b1f5-fabeb080806a\") "
Sep 09 10:44:20.891083 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:20.891052    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fed0dea-cb39-4a3a-b1f5-fabeb080806a-kube-api-access-sdm6r" (OuterVolumeSpecName: "kube-api-access-sdm6r") pod "5fed0dea-cb39-4a3a-b1f5-fabeb080806a" (UID: "5fed0dea-cb39-4a3a-b1f5-fabeb080806a"). InnerVolumeSpecName "kube-api-access-sdm6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:44:20.898735 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:20.898710    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fed0dea-cb39-4a3a-b1f5-fabeb080806a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5fed0dea-cb39-4a3a-b1f5-fabeb080806a" (UID: "5fed0dea-cb39-4a3a-b1f5-fabeb080806a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:44:20.990028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:20.989996    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdm6r\" (UniqueName: \"kubernetes.io/projected/5fed0dea-cb39-4a3a-b1f5-fabeb080806a-kube-api-access-sdm6r\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:44:20.990028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:20.990023    2570 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5fed0dea-cb39-4a3a-b1f5-fabeb080806a-tls-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:44:21.111267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:21.111234    2570 generic.go:356] "Generic (PLEG): container finished" podID="5fed0dea-cb39-4a3a-b1f5-fabeb080806a" containerID="a00691e141ff5b1688cffa5c156d6bc936d07158ec5b70621f1efdfc837a974d" exitCode=0
Sep 09 10:44:21.111410 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:21.111280    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7889699bc-2l66p"
Sep 09 10:44:21.111410 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:21.111315    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7889699bc-2l66p" event={"ID":"5fed0dea-cb39-4a3a-b1f5-fabeb080806a","Type":"ContainerDied","Data":"a00691e141ff5b1688cffa5c156d6bc936d07158ec5b70621f1efdfc837a974d"}
Sep 09 10:44:21.111410 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:21.111358    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7889699bc-2l66p" event={"ID":"5fed0dea-cb39-4a3a-b1f5-fabeb080806a","Type":"ContainerDied","Data":"79a1227e48561bc80a148b3de151b2445a6437b056f9c3dc3a87ef6b93bf0124"}
Sep 09 10:44:21.111410 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:21.111377    2570 scope.go:117] "RemoveContainer" containerID="a00691e141ff5b1688cffa5c156d6bc936d07158ec5b70621f1efdfc837a974d"
Sep 09 10:44:21.120556 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:21.120539    2570 scope.go:117] "RemoveContainer" containerID="a00691e141ff5b1688cffa5c156d6bc936d07158ec5b70621f1efdfc837a974d"
Sep 09 10:44:21.120873 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:44:21.120822    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00691e141ff5b1688cffa5c156d6bc936d07158ec5b70621f1efdfc837a974d\": container with ID starting with a00691e141ff5b1688cffa5c156d6bc936d07158ec5b70621f1efdfc837a974d not found: ID does not exist" containerID="a00691e141ff5b1688cffa5c156d6bc936d07158ec5b70621f1efdfc837a974d"
Sep 09 10:44:21.120947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:21.120885    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00691e141ff5b1688cffa5c156d6bc936d07158ec5b70621f1efdfc837a974d"} err="failed to get container status \"a00691e141ff5b1688cffa5c156d6bc936d07158ec5b70621f1efdfc837a974d\": rpc error: code = NotFound desc = could not find container \"a00691e141ff5b1688cffa5c156d6bc936d07158ec5b70621f1efdfc837a974d\": container with ID starting with a00691e141ff5b1688cffa5c156d6bc936d07158ec5b70621f1efdfc837a974d not found: ID does not exist"
Sep 09 10:44:21.146849 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:21.146796    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7889699bc-2l66p"]
Sep 09 10:44:21.158500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:21.158476    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7889699bc-2l66p"]
Sep 09 10:44:21.163386 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:21.163367    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:21.256285 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:21.256260    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:22.468495 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:22.468444    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:22.568036 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:22.568008    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:22.844274 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:22.844238    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fed0dea-cb39-4a3a-b1f5-fabeb080806a" path="/var/lib/kubelet/pods/5fed0dea-cb39-4a3a-b1f5-fabeb080806a/volumes"
Sep 09 10:44:37.841375 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:37.841339    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:37.939349 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:37.939311    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:38.121269 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.121166    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"]
Sep 09 10:44:38.121708 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.121691    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fed0dea-cb39-4a3a-b1f5-fabeb080806a" containerName="authorino"
Sep 09 10:44:38.121753 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.121711    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fed0dea-cb39-4a3a-b1f5-fabeb080806a" containerName="authorino"
Sep 09 10:44:38.121805 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.121783    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fed0dea-cb39-4a3a-b1f5-fabeb080806a" containerName="authorino"
Sep 09 10:44:38.125089 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.125068    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.128896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.128874    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-40f1e9-openshift-default-dockercfg-96qtd\""
Sep 09 10:44:38.138046 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.138015    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"]
Sep 09 10:44:38.154922 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.154896    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:38.244006 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.243973    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-token\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.244006 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.244004    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istiod-ca-cert\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.244256 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.244033    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-workload-socket\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.244256 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.244163    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-workload-certs\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.244256 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.244194    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-podinfo\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.244256 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.244234    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhx82\" (UniqueName: \"kubernetes.io/projected/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-kube-api-access-hhx82\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.244407 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.244259    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-credential-socket\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.244407 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.244275    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-data\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.244407 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.244324    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-envoy\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.258937 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.258910    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:38.345144 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.345111    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-workload-certs\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.345144 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.345142    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-podinfo\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.345403 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.345185    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhx82\" (UniqueName: \"kubernetes.io/projected/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-kube-api-access-hhx82\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.345403 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.345208    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-credential-socket\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.345403 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.345224    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-data\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.345403 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.345252    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-envoy\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.345403 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.345284    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-token\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.345403 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.345307    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istiod-ca-cert\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.345403 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.345343    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-workload-socket\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.345757 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.345610    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-workload-certs\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.345757 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.345665    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-data\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.345757 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.345758    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-workload-socket\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.345937 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.345858    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-credential-socket\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.346143 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.346124    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istiod-ca-cert\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.347548 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.347529    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-envoy\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.347641 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.347625    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-podinfo\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.354583 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.354560    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-token\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.355184 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.355164    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhx82\" (UniqueName: \"kubernetes.io/projected/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-kube-api-access-hhx82\") pod \"e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") " pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.436441 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.436361    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:38.720503 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.720476    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"]
Sep 09 10:44:38.722852 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:44:38.722758    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c3d931_4dfd_4d67_ba44_4f9bcb62465a.slice/crio-81374a4157af5ffde04076ccbb132ec574bc8dd71c184e5e4c1002c2c2a7c3e6 WatchSource:0}: Error finding container 81374a4157af5ffde04076ccbb132ec574bc8dd71c184e5e4c1002c2c2a7c3e6: Status 404 returned error can't find the container with id 81374a4157af5ffde04076ccbb132ec574bc8dd71c184e5e4c1002c2c2a7c3e6
Sep 09 10:44:38.725305 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.725273    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:44:38.725385 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.725342    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:44:38.725385 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:38.725373    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:44:39.194993 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:39.194960    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" event={"ID":"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a","Type":"ContainerStarted","Data":"bae3131a7cb477364193ace3607b175db328438246be745af97954abe173e5c5"}
Sep 09 10:44:39.195394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:39.195009    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" event={"ID":"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a","Type":"ContainerStarted","Data":"81374a4157af5ffde04076ccbb132ec574bc8dd71c184e5e4c1002c2c2a7c3e6"}
Sep 09 10:44:39.214391 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:39.214362    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:39.227436 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:39.227391    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" podStartSLOduration=1.227377434 podStartE2EDuration="1.227377434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:44:39.22573092 +0000 UTC m=+1118.889369895" watchObservedRunningTime="2026-09-09 10:44:39.227377434 +0000 UTC m=+1118.891016405"
Sep 09 10:44:39.309968 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:39.309934    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:39.437327 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:39.437294    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:39.442453 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:39.442428    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:39.720759 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:39.720696    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:40.199898 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:40.199872    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:40.200866 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:40.200843    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:41.179180 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:41.179148    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:41.263602 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:41.263560    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:42.330624 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.330591    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"]
Sep 09 10:44:42.334177 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.334155    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:42.337856 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.337836    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-40f1e9-serving-cert\""
Sep 09 10:44:42.350576 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.350553    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"]
Sep 09 10:44:42.394809 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.394783    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:42.488221 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.488186    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1056f9-b34f-4cb7-994f-5338cfbf7579-openshift-service-ca\") pod \"maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:42.488388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.488312    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-metrics-certs\") pod \"maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:42.488388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.488344    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v82d\" (UniqueName: \"kubernetes.io/projected/3b1056f9-b34f-4cb7-994f-5338cfbf7579-kube-api-access-7v82d\") pod \"maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:42.488388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.488366    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-maas-api-tls\") pod \"maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:42.495922 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.495880    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:42.589199 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.589111    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1056f9-b34f-4cb7-994f-5338cfbf7579-openshift-service-ca\") pod \"maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:42.589367 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.589222    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-metrics-certs\") pod \"maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:42.589367 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.589251    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v82d\" (UniqueName: \"kubernetes.io/projected/3b1056f9-b34f-4cb7-994f-5338cfbf7579-kube-api-access-7v82d\") pod \"maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:42.589367 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.589276    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-maas-api-tls\") pod \"maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:42.589536 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:44:42.589425    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-serving-cert: secret "maas-api-e2e-inf-a-40f1e9-serving-cert" not found
Sep 09 10:44:42.589536 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:44:42.589509    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-maas-api-tls podName:3b1056f9-b34f-4cb7-994f-5338cfbf7579 nodeName:}" failed. No retries permitted until 2026-09-09 10:44:43.089485999 +0000 UTC m=+1122.753124974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-maas-api-tls") pod "maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc" (UID: "3b1056f9-b34f-4cb7-994f-5338cfbf7579") : secret "maas-api-e2e-inf-a-40f1e9-serving-cert" not found
Sep 09 10:44:42.590130 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.590103    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1056f9-b34f-4cb7-994f-5338cfbf7579-openshift-service-ca\") pod \"maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:42.591857 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.591835    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-metrics-certs\") pod \"maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:42.602410 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.602387    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v82d\" (UniqueName: \"kubernetes.io/projected/3b1056f9-b34f-4cb7-994f-5338cfbf7579-kube-api-access-7v82d\") pod \"maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:42.875733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:42.875617    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:43.095148 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:43.095114    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-maas-api-tls\") pod \"maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:43.097923 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:43.097897    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-maas-api-tls\") pod \"maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:43.134974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:43.134896    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:43.243418 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:43.243389    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:43.244374 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:43.244354    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:43.519065 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:43.519042    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"]
Sep 09 10:44:43.520531 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:44:43.520507    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1056f9_b34f_4cb7_994f_5338cfbf7579.slice/crio-dc8e654686b4307f7b324bcc2b54853413a03cf820f8c624ed5ff07735e2d4a6 WatchSource:0}: Error finding container dc8e654686b4307f7b324bcc2b54853413a03cf820f8c624ed5ff07735e2d4a6: Status 404 returned error can't find the container with id dc8e654686b4307f7b324bcc2b54853413a03cf820f8c624ed5ff07735e2d4a6
Sep 09 10:44:44.219346 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:44.219311    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc" event={"ID":"3b1056f9-b34f-4cb7-994f-5338cfbf7579","Type":"ContainerStarted","Data":"dc8e654686b4307f7b324bcc2b54853413a03cf820f8c624ed5ff07735e2d4a6"}
Sep 09 10:44:45.226224 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:45.226183    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc" event={"ID":"3b1056f9-b34f-4cb7-994f-5338cfbf7579","Type":"ContainerStarted","Data":"9937beacfb2360f1c0f1b804cf3c1f24090939e63648ba77f0f67717d87020ca"}
Sep 09 10:44:45.226611 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:45.226262    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:45.247816 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:45.247753    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc" podStartSLOduration=2.51310687 podStartE2EDuration="3.247741599s" totalImagesPullingTime="734.634729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:44:45.245015488 +0000 UTC m=+1124.908654462" watchObservedRunningTime="2026-09-09 10:44:45.247741599 +0000 UTC m=+1124.911380622"
Sep 09 10:44:46.412041 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.412011    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:46.513301 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.513263    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:46.613406 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.613375    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:46.799193 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.799161    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"]
Sep 09 10:44:46.802789 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.802742    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:46.806415 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.806390    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e52855bf-openshift-default-dockercfg-wxg57\""
Sep 09 10:44:46.816616 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.816593    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"]
Sep 09 10:44:46.829342 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.829318    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:46.930067 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.930036    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-workload-certs\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:46.930067 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.930071    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-token\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:46.930233 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.930177    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istiod-ca-cert\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:46.930233 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.930213    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-credential-socket\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:46.930301 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.930239    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-data\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:46.930335 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.930303    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-workload-socket\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:46.930387 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.930372    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-envoy\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:46.930462 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.930447    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-podinfo\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:46.930526 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:46.930504    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj8d4\" (UniqueName: \"kubernetes.io/projected/6cc0911a-4a0c-4693-903e-9e30bbae22b5-kube-api-access-hj8d4\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.031348 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.031323    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-envoy\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.031476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.031364    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-podinfo\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.031476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.031386    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj8d4\" (UniqueName: \"kubernetes.io/projected/6cc0911a-4a0c-4693-903e-9e30bbae22b5-kube-api-access-hj8d4\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.031476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.031411    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-workload-certs\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.031476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.031439    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-token\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.031695 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.031495    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istiod-ca-cert\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.031695 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.031525    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-credential-socket\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.031695 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.031548    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-data\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.031695 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.031580    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-workload-socket\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.031936 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.031843    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-workload-certs\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.031936 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.031925    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-credential-socket\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.032076 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.032006    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-workload-socket\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.032158 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.032144    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-data\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.032301 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.032283    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istiod-ca-cert\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.034166 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.034143    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-envoy\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.034351 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.034334    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-podinfo\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.042750 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.042731    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-token\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.043069 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.043053    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj8d4\" (UniqueName: \"kubernetes.io/projected/6cc0911a-4a0c-4693-903e-9e30bbae22b5-kube-api-access-hj8d4\") pod \"e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") " pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.113470 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.113380    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:47.393907 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.393879    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"]
Sep 09 10:44:47.394056 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:44:47.394033    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc0911a_4a0c_4693_903e_9e30bbae22b5.slice/crio-ec7f4ca239233ecd1c5ec89c955b333e796e5cfcaaa6a44161a67e800e15f27e WatchSource:0}: Error finding container ec7f4ca239233ecd1c5ec89c955b333e796e5cfcaaa6a44161a67e800e15f27e: Status 404 returned error can't find the container with id ec7f4ca239233ecd1c5ec89c955b333e796e5cfcaaa6a44161a67e800e15f27e
Sep 09 10:44:47.396366 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.396334    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:44:47.396471 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.396399    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:44:47.396471 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.396427    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:44:47.703512 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:47.703350    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:48.241893 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:48.241860    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" event={"ID":"6cc0911a-4a0c-4693-903e-9e30bbae22b5","Type":"ContainerStarted","Data":"7c1f147cc4c7ea2d575f5d283695cde935e8cc4b7742e1c5bbb760df6699d17c"}
Sep 09 10:44:48.242065 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:48.241912    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" event={"ID":"6cc0911a-4a0c-4693-903e-9e30bbae22b5","Type":"ContainerStarted","Data":"ec7f4ca239233ecd1c5ec89c955b333e796e5cfcaaa6a44161a67e800e15f27e"}
Sep 09 10:44:48.287969 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:48.287920    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" podStartSLOduration=2.287904332 podStartE2EDuration="2.287904332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:44:48.285264891 +0000 UTC m=+1127.948903865" watchObservedRunningTime="2026-09-09 10:44:48.287904332 +0000 UTC m=+1127.951543306"
Sep 09 10:44:48.408292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:48.408261    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:48.712079 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:48.711998    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:48.871200 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:48.871162    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:49.113926 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:49.113894    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:49.118976 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:49.118951    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:49.249980 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:49.249947    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:49.259053 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:49.259019    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:44:49.782108 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:49.782077    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:49.865193 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:49.865152    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:50.990152 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:50.990125    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:51.109785 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:51.109731    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:51.236359 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:51.236334    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:44:51.504414 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:51.504384    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:51.673546 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:51.673516    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:51.773078 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:51.773005    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:51.870575 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:51.870544    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:52.053890 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:52.053670    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:53.082952 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.082919    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"]
Sep 09 10:44:53.086719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.086695    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.089526 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.089508    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-4bbfe3-serving-cert\""
Sep 09 10:44:53.100573 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.100549    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"]
Sep 09 10:44:53.133919 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.133895    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:53.201610 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.201575    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/328ffd1f-d294-4a7f-a245-3b280a9e8400-openshift-service-ca\") pod \"maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.201610 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.201616    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-maas-api-tls\") pod \"maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.201863 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.201633    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-metrics-certs\") pod \"maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.201863 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.201707    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqc42\" (UniqueName: \"kubernetes.io/projected/328ffd1f-d294-4a7f-a245-3b280a9e8400-kube-api-access-fqc42\") pod \"maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.235200 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.235157    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:53.303317 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.303289    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/328ffd1f-d294-4a7f-a245-3b280a9e8400-openshift-service-ca\") pod \"maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.303500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.303332    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-maas-api-tls\") pod \"maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.303500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.303358    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-metrics-certs\") pod \"maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.303500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.303398    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqc42\" (UniqueName: \"kubernetes.io/projected/328ffd1f-d294-4a7f-a245-3b280a9e8400-kube-api-access-fqc42\") pod \"maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.303500 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:44:53.303476    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-serving-cert: secret "maas-api-e2e-autores-4bbfe3-serving-cert" not found
Sep 09 10:44:53.303704 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:44:53.303562    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-maas-api-tls podName:328ffd1f-d294-4a7f-a245-3b280a9e8400 nodeName:}" failed. No retries permitted until 2026-09-09 10:44:53.803538444 +0000 UTC m=+1133.467177418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-maas-api-tls") pod "maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264" (UID: "328ffd1f-d294-4a7f-a245-3b280a9e8400") : secret "maas-api-e2e-autores-4bbfe3-serving-cert" not found
Sep 09 10:44:53.304172 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.304156    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/328ffd1f-d294-4a7f-a245-3b280a9e8400-openshift-service-ca\") pod \"maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.305670 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.305651    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-metrics-certs\") pod \"maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.311687 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.311669    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqc42\" (UniqueName: \"kubernetes.io/projected/328ffd1f-d294-4a7f-a245-3b280a9e8400-kube-api-access-fqc42\") pod \"maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.542414 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.542341    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:53.676467 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.676425    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:53.808991 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.808911    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-maas-api-tls\") pod \"maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.811326 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.811302    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-maas-api-tls\") pod \"maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:53.814126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.814104    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:53.985995 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.985965    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"]
Sep 09 10:44:53.990635 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.990609    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"
Sep 09 10:44:53.993542 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.993518    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-4bbfe3\""
Sep 09 10:44:53.993693 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:53.993663    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-4bbfe3-dockercfg-2rqjw\""
Sep 09 10:44:54.000479 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:54.000457    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"]
Sep 09 10:44:54.001866 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:54.001848    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:54.011405 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:54.011382    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb70e500-9155-46ab-b392-560019ad9384-plugins-config-volume\") pod \"payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6\" (UID: \"eb70e500-9155-46ab-b392-560019ad9384\") " pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"
Sep 09 10:44:54.011517 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:54.011468    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz9fh\" (UniqueName: \"kubernetes.io/projected/eb70e500-9155-46ab-b392-560019ad9384-kube-api-access-gz9fh\") pod \"payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6\" (UID: \"eb70e500-9155-46ab-b392-560019ad9384\") " pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"
Sep 09 10:44:54.113066 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:54.112889    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9fh\" (UniqueName: \"kubernetes.io/projected/eb70e500-9155-46ab-b392-560019ad9384-kube-api-access-gz9fh\") pod \"payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6\" (UID: \"eb70e500-9155-46ab-b392-560019ad9384\") " pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"
Sep 09 10:44:54.113066 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:54.112981    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb70e500-9155-46ab-b392-560019ad9384-plugins-config-volume\") pod \"payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6\" (UID: \"eb70e500-9155-46ab-b392-560019ad9384\") " pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"
Sep 09 10:44:54.113737 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:54.113716    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb70e500-9155-46ab-b392-560019ad9384-plugins-config-volume\") pod \"payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6\" (UID: \"eb70e500-9155-46ab-b392-560019ad9384\") " pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"
Sep 09 10:44:54.122920 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:54.122895    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz9fh\" (UniqueName: \"kubernetes.io/projected/eb70e500-9155-46ab-b392-560019ad9384-kube-api-access-gz9fh\") pod \"payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6\" (UID: \"eb70e500-9155-46ab-b392-560019ad9384\") " pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"
Sep 09 10:44:54.301888 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:54.301858    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"
Sep 09 10:44:54.304551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:54.304525    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"]
Sep 09 10:44:54.306273 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:44:54.306185    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328ffd1f_d294_4a7f_a245_3b280a9e8400.slice/crio-a12f5bedb8bff9c5b37d0fdc8e17d25c675cfdac4b88232f9012c6b3ac118a4c WatchSource:0}: Error finding container a12f5bedb8bff9c5b37d0fdc8e17d25c675cfdac4b88232f9012c6b3ac118a4c: Status 404 returned error can't find the container with id a12f5bedb8bff9c5b37d0fdc8e17d25c675cfdac4b88232f9012c6b3ac118a4c
Sep 09 10:44:54.599617 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:54.599591    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"]
Sep 09 10:44:54.601914 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:44:54.601883    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb70e500_9155_46ab_b392_560019ad9384.slice/crio-e7ce11f12db651250583bd1142624a4caf5b2b4b3e7a46a7c4ba93af94e3a306 WatchSource:0}: Error finding container e7ce11f12db651250583bd1142624a4caf5b2b4b3e7a46a7c4ba93af94e3a306: Status 404 returned error can't find the container with id e7ce11f12db651250583bd1142624a4caf5b2b4b3e7a46a7c4ba93af94e3a306
Sep 09 10:44:55.286921 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:55.286882    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6" event={"ID":"eb70e500-9155-46ab-b392-560019ad9384","Type":"ContainerStarted","Data":"3a297d23753592b023658d906a76335522d6fbd6dd8db2670391bf9db56bfe72"}
Sep 09 10:44:55.287375 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:55.286945    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6" event={"ID":"eb70e500-9155-46ab-b392-560019ad9384","Type":"ContainerStarted","Data":"e7ce11f12db651250583bd1142624a4caf5b2b4b3e7a46a7c4ba93af94e3a306"}
Sep 09 10:44:55.287375 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:55.287080    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"
Sep 09 10:44:55.288526 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:55.288504    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264" event={"ID":"328ffd1f-d294-4a7f-a245-3b280a9e8400","Type":"ContainerStarted","Data":"71cc44152c6f793cafa7b15fa64501dec7500f7ff0156904b6ff0add1adebb4f"}
Sep 09 10:44:55.288661 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:55.288543    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264" event={"ID":"328ffd1f-d294-4a7f-a245-3b280a9e8400","Type":"ContainerStarted","Data":"a12f5bedb8bff9c5b37d0fdc8e17d25c675cfdac4b88232f9012c6b3ac118a4c"}
Sep 09 10:44:55.288661 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:55.288569    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:44:55.308757 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:55.308698    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6" podStartSLOduration=2.308678907 podStartE2EDuration="2.308678907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:44:55.307848716 +0000 UTC m=+1134.971487691" watchObservedRunningTime="2026-09-09 10:44:55.308678907 +0000 UTC m=+1134.972317882"
Sep 09 10:44:55.331363 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:55.331304    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264" podStartSLOduration=2.0103235 podStartE2EDuration="2.331285449s" totalImagesPullingTime="320.961949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:44:55.330803537 +0000 UTC m=+1134.994442513" watchObservedRunningTime="2026-09-09 10:44:55.331285449 +0000 UTC m=+1134.994924427"
Sep 09 10:44:56.241905 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:56.241870    2570 patch_prober.go:28] interesting pod/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 10:44:56.242182 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:56.241935    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 10:44:56.476696 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:56.476654    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:56.622192 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:56.622158    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:56.719469 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:56.719437    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:56.819645 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:56.819563    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:56.917963 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:56.917934    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:57.097993 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:57.097913    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:44:58.305738 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:58.305710    2570 generic.go:356] "Generic (PLEG): container finished" podID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerID="bae3131a7cb477364193ace3607b175db328438246be745af97954abe173e5c5" exitCode=0
Sep 09 10:44:58.306163 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:58.305799    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" event={"ID":"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a","Type":"ContainerDied","Data":"bae3131a7cb477364193ace3607b175db328438246be745af97954abe173e5c5"}
Sep 09 10:44:58.306214 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:58.306205    2570 scope.go:117] "RemoveContainer" containerID="bae3131a7cb477364193ace3607b175db328438246be745af97954abe173e5c5"
Sep 09 10:44:58.307283 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:58.307257    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:44:58.307349 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:58.307335    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:44:58.307393 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:58.307383    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:44:59.312116 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:59.312081    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" event={"ID":"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a","Type":"ContainerStarted","Data":"d5803b8d363d42c16cea2483aa808f6c9102c63165b07a4b00a094f537d0c209"}
Sep 09 10:44:59.437680 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:59.437588    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:44:59.437866 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:59.437831    2570 patch_prober.go:28] interesting pod/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 09 10:44:59.437944 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:59.437887    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 09 10:44:59.568008 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:44:59.567975    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:00.423934 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:00.423895    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:00.437040 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:00.437005    2570 patch_prober.go:28] interesting pod/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 09 10:45:00.437203 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:00.437057    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 09 10:45:01.299392 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:01.299363    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:45:01.349610 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:01.349579    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:01.437555 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:01.437517    2570 patch_prober.go:28] interesting pod/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 09 10:45:01.437960 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:01.437577    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 09 10:45:02.437210 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:02.437174    2570 patch_prober.go:28] interesting pod/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 09 10:45:02.437373 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:02.437244    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 09 10:45:03.437586 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:03.437555    2570 patch_prober.go:28] interesting pod/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 09 10:45:03.437972 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:03.437611    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 09 10:45:04.437288 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:04.437204    2570 patch_prober.go:28] interesting pod/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 09 10:45:04.437288 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:04.437273    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 09 10:45:04.659430 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:04.659359    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:04.760534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:04.760488    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:05.290954 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:05.290912    2570 patch_prober.go:28] interesting pod/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 10:45:05.291133 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:05.290963    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 10:45:05.597755 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:05.597661    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:45:05.598530 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:05.598503    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:45:05.599390 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:05.599370    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:45:06.155626 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:06.155597    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:06.296300 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:06.296274    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"
Sep 09 10:45:06.462089 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:06.462010    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:06.804904 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:06.804858    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:07.367631 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:07.367599    2570 generic.go:356] "Generic (PLEG): container finished" podID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerID="7c1f147cc4c7ea2d575f5d283695cde935e8cc4b7742e1c5bbb760df6699d17c" exitCode=0
Sep 09 10:45:07.368072 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:07.367674    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" event={"ID":"6cc0911a-4a0c-4693-903e-9e30bbae22b5","Type":"ContainerDied","Data":"7c1f147cc4c7ea2d575f5d283695cde935e8cc4b7742e1c5bbb760df6699d17c"}
Sep 09 10:45:07.368327 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:07.368304    2570 scope.go:117] "RemoveContainer" containerID="7c1f147cc4c7ea2d575f5d283695cde935e8cc4b7742e1c5bbb760df6699d17c"
Sep 09 10:45:07.369409 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:07.369378    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:07.369739 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:07.369451    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:07.369739 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:07.369482    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:07.406523 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:07.406454    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:08.376380 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:08.376343    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" event={"ID":"6cc0911a-4a0c-4693-903e-9e30bbae22b5","Type":"ContainerStarted","Data":"17729f2111c0c239c512c43abc7db845e812290ae02d19a87f61a7864c5e2751"}
Sep 09 10:45:09.114032 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:09.113999    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:45:09.114286 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:09.114255    2570 patch_prober.go:28] interesting pod/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 10:45:09.114332 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:09.114301    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 10:45:09.828554 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:09.828513    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:09.917324 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:09.917288    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:10.114593 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:10.114510    2570 patch_prober.go:28] interesting pod/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 10:45:10.114593 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:10.114574    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 10:45:11.114057 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:11.114017    2570 patch_prober.go:28] interesting pod/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 10:45:11.114507 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:11.114085    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 10:45:11.177107 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:11.177073    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:11.274580 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:11.274546    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:11.456677 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:11.456578    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:11.552163 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:11.552135    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:11.893601 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:11.893565    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:12.035751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:12.035721    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:12.114550 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:12.114512    2570 patch_prober.go:28] interesting pod/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 10:45:12.114948 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:12.114572    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 10:45:13.113872 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:13.113804    2570 patch_prober.go:28] interesting pod/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 10:45:13.114082 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:13.113912    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 10:45:13.165463 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:13.165428    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:14.117836 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:14.117805    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:45:14.118087 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:14.118068    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:45:14.118619 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:14.118601    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:45:14.626623 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:14.626591    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:15.426498 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:15.426466    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:16.201203 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:16.201164    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:16.664493 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:16.664462    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:21.216106 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:21.216070    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:21.829982 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:21.829953    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:21.917739 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:21.917702    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:21.990168 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:21.990142    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:22.077031 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:22.077006    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:22.153289 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:22.153213    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:22.229402 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:22.229372    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"]
Sep 09 10:45:22.229872 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:22.229609    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerName="istio-proxy" containerID="cri-o://17729f2111c0c239c512c43abc7db845e812290ae02d19a87f61a7864c5e2751" gracePeriod=30
Sep 09 10:45:22.233354 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:22.233324    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:22.717684 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:22.717651    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:22.812129 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:22.812099    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:22.943219 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:22.943185    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:23.211590 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.211522    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"]
Sep 09 10:45:23.214835 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.214815    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.217550 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.217530    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-92ac8218-gw-openshift-default-dockercfg-845cf\""
Sep 09 10:45:23.232981 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.232951    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istiod-ca-cert\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.233548 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.233402    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-envoy\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.233699 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.233682    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-credential-socket\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.233892 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.233874    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8rtl\" (UniqueName: \"kubernetes.io/projected/e76597ba-f9ca-424e-8d2d-b5e3434fa026-kube-api-access-d8rtl\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.234070 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.234054    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-podinfo\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.234225 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.234209    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-token\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.234422 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.234392    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-data\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.234623 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.234592    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-workload-socket\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.234822 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.234806    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-workload-certs\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.235085 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.235069    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"]
Sep 09 10:45:23.247965 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.247937    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:23.335717 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.335685    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-workload-certs\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.335930 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.335736    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istiod-ca-cert\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.335930 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.335821    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-envoy\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.335930 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.335877    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-credential-socket\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.335930 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.335911    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8rtl\" (UniqueName: \"kubernetes.io/projected/e76597ba-f9ca-424e-8d2d-b5e3434fa026-kube-api-access-d8rtl\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.336167 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.335956    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-podinfo\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.336167 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.336004    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-token\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.336167 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.336082    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-data\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.336167 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.336112    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-workload-socket\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.336167 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.336135    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-workload-certs\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.336433 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.336308    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-credential-socket\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.336485 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.336463    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-data\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.336548 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.336486    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-workload-socket\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.336601 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.336554    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istiod-ca-cert\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.338337 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.338312    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-envoy\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.338452 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.338382    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-podinfo\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.344842 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.344808    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8rtl\" (UniqueName: \"kubernetes.io/projected/e76597ba-f9ca-424e-8d2d-b5e3434fa026-kube-api-access-d8rtl\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.345097 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.345079    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-token\") pod \"e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") " pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.526450 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.526422    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:23.576738 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.576692    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:23.800408 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.800385    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"]
Sep 09 10:45:23.801865 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:45:23.801835    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76597ba_f9ca_424e_8d2d_b5e3434fa026.slice/crio-e2ab5682936ef81cf72aa4177e5f19097677c44003ebede9b55ace197da9147e WatchSource:0}: Error finding container e2ab5682936ef81cf72aa4177e5f19097677c44003ebede9b55ace197da9147e: Status 404 returned error can't find the container with id e2ab5682936ef81cf72aa4177e5f19097677c44003ebede9b55ace197da9147e
Sep 09 10:45:23.804046 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.804011    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:23.804119 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.804077    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:23.804119 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:23.804114    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:24.460241 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:24.460207    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5" event={"ID":"e76597ba-f9ca-424e-8d2d-b5e3434fa026","Type":"ContainerStarted","Data":"4622c5f62b5ae8e150fb098b2a0777776c405e7749bef0dc07a41f9c237234ed"}
Sep 09 10:45:24.460602 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:24.460257    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5" event={"ID":"e76597ba-f9ca-424e-8d2d-b5e3434fa026","Type":"ContainerStarted","Data":"e2ab5682936ef81cf72aa4177e5f19097677c44003ebede9b55ace197da9147e"}
Sep 09 10:45:24.483638 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:24.483585    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5" podStartSLOduration=1.483572352 podStartE2EDuration="1.483572352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:24.481506595 +0000 UTC m=+1164.145145568" watchObservedRunningTime="2026-09-09 10:45:24.483572352 +0000 UTC m=+1164.147211325"
Sep 09 10:45:24.527341 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:24.527318    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:24.532407 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:24.532377    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:25.466068 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:25.466037    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:25.467084 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:25.467060    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:25.961144 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:25.961111    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:26.064091 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.064059    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:26.109821 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.109791    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"]
Sep 09 10:45:26.120895 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.120866    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.124263 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.124240    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-92ac8218-serving-cert\""
Sep 09 10:45:26.130980 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.130957    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"]
Sep 09 10:45:26.159065 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.159036    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:26.162317 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.162294    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea283524-3f7a-4860-9990-b2cf38296a85-maas-api-tls\") pod \"maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.162441 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.162343    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea283524-3f7a-4860-9990-b2cf38296a85-openshift-service-ca\") pod \"maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.162503 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.162452    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea283524-3f7a-4860-9990-b2cf38296a85-metrics-certs\") pod \"maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.162559 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.162498    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgjtr\" (UniqueName: \"kubernetes.io/projected/ea283524-3f7a-4860-9990-b2cf38296a85-kube-api-access-wgjtr\") pod \"maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.263127 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.263093    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea283524-3f7a-4860-9990-b2cf38296a85-metrics-certs\") pod \"maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.263274 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.263138    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgjtr\" (UniqueName: \"kubernetes.io/projected/ea283524-3f7a-4860-9990-b2cf38296a85-kube-api-access-wgjtr\") pod \"maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.263274 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.263211    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea283524-3f7a-4860-9990-b2cf38296a85-maas-api-tls\") pod \"maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.263274 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.263261    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea283524-3f7a-4860-9990-b2cf38296a85-openshift-service-ca\") pod \"maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.263912 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.263889    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea283524-3f7a-4860-9990-b2cf38296a85-openshift-service-ca\") pod \"maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.265446 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.265423    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea283524-3f7a-4860-9990-b2cf38296a85-maas-api-tls\") pod \"maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.265654 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.265632    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea283524-3f7a-4860-9990-b2cf38296a85-metrics-certs\") pod \"maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.282796 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.282758    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgjtr\" (UniqueName: \"kubernetes.io/projected/ea283524-3f7a-4860-9990-b2cf38296a85-kube-api-access-wgjtr\") pod \"maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.434239 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.434211    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:26.710542 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:26.710519    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"]
Sep 09 10:45:26.712643 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:45:26.712621    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea283524_3f7a_4860_9990_b2cf38296a85.slice/crio-f41eb61234b86a75b8d3acd2aeea62385211caa21dca19ed1252ff4d5ea2c0c9 WatchSource:0}: Error finding container f41eb61234b86a75b8d3acd2aeea62385211caa21dca19ed1252ff4d5ea2c0c9: Status 404 returned error can't find the container with id f41eb61234b86a75b8d3acd2aeea62385211caa21dca19ed1252ff4d5ea2c0c9
Sep 09 10:45:27.081624 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.081591    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:27.183124 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.183083    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:27.477960 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.477931    2570 generic.go:356] "Generic (PLEG): container finished" podID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerID="17729f2111c0c239c512c43abc7db845e812290ae02d19a87f61a7864c5e2751" exitCode=0
Sep 09 10:45:27.478090 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.478000    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" event={"ID":"6cc0911a-4a0c-4693-903e-9e30bbae22b5","Type":"ContainerDied","Data":"17729f2111c0c239c512c43abc7db845e812290ae02d19a87f61a7864c5e2751"}
Sep 09 10:45:27.478090 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.478056    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4" event={"ID":"6cc0911a-4a0c-4693-903e-9e30bbae22b5","Type":"ContainerDied","Data":"ec7f4ca239233ecd1c5ec89c955b333e796e5cfcaaa6a44161a67e800e15f27e"}
Sep 09 10:45:27.478090 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.478074    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec7f4ca239233ecd1c5ec89c955b333e796e5cfcaaa6a44161a67e800e15f27e"
Sep 09 10:45:27.478090 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.478089    2570 scope.go:117] "RemoveContainer" containerID="7c1f147cc4c7ea2d575f5d283695cde935e8cc4b7742e1c5bbb760df6699d17c"
Sep 09 10:45:27.479823 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.479798    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq" event={"ID":"ea283524-3f7a-4860-9990-b2cf38296a85","Type":"ContainerStarted","Data":"eac5276a37b9d179cb911a8a4bf0ebbf8ebcc300dff51126e30f241c46e06a33"}
Sep 09 10:45:27.479955 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.479848    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq" event={"ID":"ea283524-3f7a-4860-9990-b2cf38296a85","Type":"ContainerStarted","Data":"f41eb61234b86a75b8d3acd2aeea62385211caa21dca19ed1252ff4d5ea2c0c9"}
Sep 09 10:45:27.482672 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.482655    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:45:27.520518 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.520471    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq" podStartSLOduration=1.126376142 podStartE2EDuration="1.520439955s" totalImagesPullingTime="394.063813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:27.519429134 +0000 UTC m=+1167.183068132" watchObservedRunningTime="2026-09-09 10:45:27.520439955 +0000 UTC m=+1167.184078941"
Sep 09 10:45:27.575585 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.575544    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-workload-certs\") pod \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") "
Sep 09 10:45:27.575585 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.575578    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-envoy\") pod \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") "
Sep 09 10:45:27.575837 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.575602    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istiod-ca-cert\") pod \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") "
Sep 09 10:45:27.575837 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.575628    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-workload-socket\") pod \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") "
Sep 09 10:45:27.575837 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.575658    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-token\") pod \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") "
Sep 09 10:45:27.575837 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.575810    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-credential-socket\") pod \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") "
Sep 09 10:45:27.576053 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.575867    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj8d4\" (UniqueName: \"kubernetes.io/projected/6cc0911a-4a0c-4693-903e-9e30bbae22b5-kube-api-access-hj8d4\") pod \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") "
Sep 09 10:45:27.576053 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.575872    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6cc0911a-4a0c-4693-903e-9e30bbae22b5" (UID: "6cc0911a-4a0c-4693-903e-9e30bbae22b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:45:27.576053 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.575987    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-podinfo\") pod \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") "
Sep 09 10:45:27.576053 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.576013    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-data\") pod \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\" (UID: \"6cc0911a-4a0c-4693-903e-9e30bbae22b5\") "
Sep 09 10:45:27.576261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.576022    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6cc0911a-4a0c-4693-903e-9e30bbae22b5" (UID: "6cc0911a-4a0c-4693-903e-9e30bbae22b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:45:27.576261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.576076    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6cc0911a-4a0c-4693-903e-9e30bbae22b5" (UID: "6cc0911a-4a0c-4693-903e-9e30bbae22b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:45:27.576261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.576080    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6cc0911a-4a0c-4693-903e-9e30bbae22b5" (UID: "6cc0911a-4a0c-4693-903e-9e30bbae22b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:45:27.576421 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.576329    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-data" (OuterVolumeSpecName: "istio-data") pod "6cc0911a-4a0c-4693-903e-9e30bbae22b5" (UID: "6cc0911a-4a0c-4693-903e-9e30bbae22b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:45:27.576532 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.576513    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-data\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:45:27.576600 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.576537    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-workload-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:45:27.576600 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.576552    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istiod-ca-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:45:27.576600 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.576566    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-workload-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:45:27.576600 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.576580    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-credential-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:45:27.578825 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.578796    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6cc0911a-4a0c-4693-903e-9e30bbae22b5" (UID: "6cc0911a-4a0c-4693-903e-9e30bbae22b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:45:27.578955 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.578938    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "6cc0911a-4a0c-4693-903e-9e30bbae22b5" (UID: "6cc0911a-4a0c-4693-903e-9e30bbae22b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:45:27.579172 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.579142    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc0911a-4a0c-4693-903e-9e30bbae22b5-kube-api-access-hj8d4" (OuterVolumeSpecName: "kube-api-access-hj8d4") pod "6cc0911a-4a0c-4693-903e-9e30bbae22b5" (UID: "6cc0911a-4a0c-4693-903e-9e30bbae22b5"). InnerVolumeSpecName "kube-api-access-hj8d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:45:27.579416 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.579397    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6cc0911a-4a0c-4693-903e-9e30bbae22b5" (UID: "6cc0911a-4a0c-4693-903e-9e30bbae22b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:45:27.678165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.678087    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-podinfo\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:45:27.678165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.678110    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-envoy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:45:27.678165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.678121    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cc0911a-4a0c-4693-903e-9e30bbae22b5-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:45:27.678165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:27.678130    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj8d4\" (UniqueName: \"kubernetes.io/projected/6cc0911a-4a0c-4693-903e-9e30bbae22b5-kube-api-access-hj8d4\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:45:28.485961 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:28.485931    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"
Sep 09 10:45:28.486398 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:28.486332    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:28.535483 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:28.535456    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"]
Sep 09 10:45:28.540021 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:28.539996    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e52855bf-openshift-default-7cb7b4bd6d-cwwd4"]
Sep 09 10:45:28.839911 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:28.839882    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" path="/var/lib/kubelet/pods/6cc0911a-4a0c-4693-903e-9e30bbae22b5/volumes"
Sep 09 10:45:32.458464 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:32.458430    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:34.497296 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:34.497266    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:34.759395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:34.759322    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:42.623713 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:42.623677    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:46.583990 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:46.583956    2570 generic.go:356] "Generic (PLEG): container finished" podID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerID="4622c5f62b5ae8e150fb098b2a0777776c405e7749bef0dc07a41f9c237234ed" exitCode=0
Sep 09 10:45:46.584401 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:46.584000    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5" event={"ID":"e76597ba-f9ca-424e-8d2d-b5e3434fa026","Type":"ContainerDied","Data":"4622c5f62b5ae8e150fb098b2a0777776c405e7749bef0dc07a41f9c237234ed"}
Sep 09 10:45:46.584443 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:46.584406    2570 scope.go:117] "RemoveContainer" containerID="4622c5f62b5ae8e150fb098b2a0777776c405e7749bef0dc07a41f9c237234ed"
Sep 09 10:45:46.585562 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:46.585531    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:46.585681 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:46.585610    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:46.585681 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:46.585661    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:47.349522 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:47.349471    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:47.591153 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:47.591122    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5" event={"ID":"e76597ba-f9ca-424e-8d2d-b5e3434fa026","Type":"ContainerStarted","Data":"9279066163d134447d560077a25b6fc690741c0667b7103412ec966f2f87d263"}
Sep 09 10:45:48.527277 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:48.527244    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:48.527452 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:48.527418    2570 patch_prober.go:28] interesting pod/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 10:45:48.527497 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:48.527462    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5" podUID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 10:45:49.526822 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:49.526789    2570 patch_prober.go:28] interesting pod/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 10:45:49.527194 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:49.526844    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5" podUID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 10:45:50.453280 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:50.453244    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:50.527211 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:50.527178    2570 patch_prober.go:28] interesting pod/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 10:45:50.527561 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:50.527239    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5" podUID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 10:45:51.526929 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:51.526896    2570 patch_prober.go:28] interesting pod/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 10:45:51.527081 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:51.526964    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5" podUID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 10:45:52.527786 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:52.527741    2570 patch_prober.go:28] interesting pod/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 10:45:52.528267 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:52.527836    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5" podUID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 10:45:53.531164 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:53.531137    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:53.531622 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:53.531508    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:53.532214 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:53.532198    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:45:57.573316 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.573273    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"]
Sep 09 10:45:57.573801 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.573500    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq" podUID="ea283524-3f7a-4860-9990-b2cf38296a85" containerName="maas-api" containerID="cri-o://eac5276a37b9d179cb911a8a4bf0ebbf8ebcc300dff51126e30f241c46e06a33" gracePeriod=30
Sep 09 10:45:57.613671 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.613643    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:57.714603 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.714566    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:57.814012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.813983    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:57.843043 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.843020    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:57.896673 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.896648    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea283524-3f7a-4860-9990-b2cf38296a85-maas-api-tls\") pod \"ea283524-3f7a-4860-9990-b2cf38296a85\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") "
Sep 09 10:45:57.896839 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.896680    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea283524-3f7a-4860-9990-b2cf38296a85-metrics-certs\") pod \"ea283524-3f7a-4860-9990-b2cf38296a85\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") "
Sep 09 10:45:57.896839 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.896710    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea283524-3f7a-4860-9990-b2cf38296a85-openshift-service-ca\") pod \"ea283524-3f7a-4860-9990-b2cf38296a85\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") "
Sep 09 10:45:57.896839 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.896746    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgjtr\" (UniqueName: \"kubernetes.io/projected/ea283524-3f7a-4860-9990-b2cf38296a85-kube-api-access-wgjtr\") pod \"ea283524-3f7a-4860-9990-b2cf38296a85\" (UID: \"ea283524-3f7a-4860-9990-b2cf38296a85\") "
Sep 09 10:45:57.897130 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.897102    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea283524-3f7a-4860-9990-b2cf38296a85-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea283524-3f7a-4860-9990-b2cf38296a85" (UID: "ea283524-3f7a-4860-9990-b2cf38296a85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:45:57.898898 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.898877    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea283524-3f7a-4860-9990-b2cf38296a85-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ea283524-3f7a-4860-9990-b2cf38296a85" (UID: "ea283524-3f7a-4860-9990-b2cf38296a85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:45:57.899278 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.899256    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea283524-3f7a-4860-9990-b2cf38296a85-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea283524-3f7a-4860-9990-b2cf38296a85" (UID: "ea283524-3f7a-4860-9990-b2cf38296a85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:45:57.899331 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.899269    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea283524-3f7a-4860-9990-b2cf38296a85-kube-api-access-wgjtr" (OuterVolumeSpecName: "kube-api-access-wgjtr") pod "ea283524-3f7a-4860-9990-b2cf38296a85" (UID: "ea283524-3f7a-4860-9990-b2cf38296a85"). InnerVolumeSpecName "kube-api-access-wgjtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:45:57.916054 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.916030    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:57.957293 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.957264    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"]
Sep 09 10:45:57.957494 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.957474    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5" podUID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerName="istio-proxy" containerID="cri-o://9279066163d134447d560077a25b6fc690741c0667b7103412ec966f2f87d263" gracePeriod=30
Sep 09 10:45:57.985961 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.985931    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:57.997581 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.997562    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea283524-3f7a-4860-9990-b2cf38296a85-maas-api-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:45:57.997581 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.997579    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea283524-3f7a-4860-9990-b2cf38296a85-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:45:57.997733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.997589    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea283524-3f7a-4860-9990-b2cf38296a85-openshift-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:45:57.997733 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:57.997597    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgjtr\" (UniqueName: \"kubernetes.io/projected/ea283524-3f7a-4860-9990-b2cf38296a85-kube-api-access-wgjtr\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:45:58.389326 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.389290    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:58.651896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.651806    2570 generic.go:356] "Generic (PLEG): container finished" podID="ea283524-3f7a-4860-9990-b2cf38296a85" containerID="eac5276a37b9d179cb911a8a4bf0ebbf8ebcc300dff51126e30f241c46e06a33" exitCode=0
Sep 09 10:45:58.651896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.651861    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq" event={"ID":"ea283524-3f7a-4860-9990-b2cf38296a85","Type":"ContainerDied","Data":"eac5276a37b9d179cb911a8a4bf0ebbf8ebcc300dff51126e30f241c46e06a33"}
Sep 09 10:45:58.651896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.651884    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"
Sep 09 10:45:58.652462 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.651933    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq" event={"ID":"ea283524-3f7a-4860-9990-b2cf38296a85","Type":"ContainerDied","Data":"f41eb61234b86a75b8d3acd2aeea62385211caa21dca19ed1252ff4d5ea2c0c9"}
Sep 09 10:45:58.652462 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.651967    2570 scope.go:117] "RemoveContainer" containerID="eac5276a37b9d179cb911a8a4bf0ebbf8ebcc300dff51126e30f241c46e06a33"
Sep 09 10:45:58.662946 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.662928    2570 scope.go:117] "RemoveContainer" containerID="eac5276a37b9d179cb911a8a4bf0ebbf8ebcc300dff51126e30f241c46e06a33"
Sep 09 10:45:58.663228 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:45:58.663206    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac5276a37b9d179cb911a8a4bf0ebbf8ebcc300dff51126e30f241c46e06a33\": container with ID starting with eac5276a37b9d179cb911a8a4bf0ebbf8ebcc300dff51126e30f241c46e06a33 not found: ID does not exist" containerID="eac5276a37b9d179cb911a8a4bf0ebbf8ebcc300dff51126e30f241c46e06a33"
Sep 09 10:45:58.663323 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.663238    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac5276a37b9d179cb911a8a4bf0ebbf8ebcc300dff51126e30f241c46e06a33"} err="failed to get container status \"eac5276a37b9d179cb911a8a4bf0ebbf8ebcc300dff51126e30f241c46e06a33\": rpc error: code = NotFound desc = could not find container \"eac5276a37b9d179cb911a8a4bf0ebbf8ebcc300dff51126e30f241c46e06a33\": container with ID starting with eac5276a37b9d179cb911a8a4bf0ebbf8ebcc300dff51126e30f241c46e06a33 not found: ID does not exist"
Sep 09 10:45:58.668518 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.668494    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"]
Sep 09 10:45:58.668909 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.668886    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerName="istio-proxy"
Sep 09 10:45:58.668909 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.668911    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerName="istio-proxy"
Sep 09 10:45:58.668909 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.668919    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea283524-3f7a-4860-9990-b2cf38296a85" containerName="maas-api"
Sep 09 10:45:58.669050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.668924    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea283524-3f7a-4860-9990-b2cf38296a85" containerName="maas-api"
Sep 09 10:45:58.669050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.668938    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerName="istio-proxy"
Sep 09 10:45:58.669050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.668943    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerName="istio-proxy"
Sep 09 10:45:58.669050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.669001    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerName="istio-proxy"
Sep 09 10:45:58.669050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.669010    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc0911a-4a0c-4693-903e-9e30bbae22b5" containerName="istio-proxy"
Sep 09 10:45:58.669050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.669016    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea283524-3f7a-4860-9990-b2cf38296a85" containerName="maas-api"
Sep 09 10:45:58.671529 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.671511    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.673979 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.673960    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-dfb9af4b-openshift-default-dockercfg-hlbkt\""
Sep 09 10:45:58.678817 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.678797    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"]
Sep 09 10:45:58.684636 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.684617    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-92ac8218-69fc77c64d-xd2fq"]
Sep 09 10:45:58.687365 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.687347    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"]
Sep 09 10:45:58.691715 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.691665    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:58.704421 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.704390    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-token\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.704523 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.704428    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tqlt\" (UniqueName: \"kubernetes.io/projected/7de794cb-81be-4600-b724-e4ff6193ae3b-kube-api-access-5tqlt\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.704523 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.704471    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de794cb-81be-4600-b724-e4ff6193ae3b-istiod-ca-cert\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.704647 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.704525    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-envoy\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.704647 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.704604    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-credential-socket\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.704754 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.704715    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-workload-socket\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.704754 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.704746    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-podinfo\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.704871 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.704845    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-workload-certs\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.704914 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.704880    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-data\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.805493 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.805462    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-workload-socket\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.805493 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.805495    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-podinfo\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.805751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.805538    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-workload-certs\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.805751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.805562    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-data\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.805751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.805586    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-token\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.805751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.805603    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tqlt\" (UniqueName: \"kubernetes.io/projected/7de794cb-81be-4600-b724-e4ff6193ae3b-kube-api-access-5tqlt\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.805751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.805639    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de794cb-81be-4600-b724-e4ff6193ae3b-istiod-ca-cert\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.805751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.805671    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-envoy\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.805751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.805711    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-credential-socket\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.806141 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.806111    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-workload-certs\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.806195 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.806135    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-credential-socket\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.806487 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.806312    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-data\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.806487 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.806446    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-workload-socket\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.806487 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.806469    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de794cb-81be-4600-b724-e4ff6193ae3b-istiod-ca-cert\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.808338 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.808316    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-envoy\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.808338 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.808325    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-podinfo\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.813741 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.813723    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tqlt\" (UniqueName: \"kubernetes.io/projected/7de794cb-81be-4600-b724-e4ff6193ae3b-kube-api-access-5tqlt\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.813826 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.813812    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-token\") pod \"e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") " pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:58.845656 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.841078    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea283524-3f7a-4860-9990-b2cf38296a85" path="/var/lib/kubelet/pods/ea283524-3f7a-4860-9990-b2cf38296a85/volumes"
Sep 09 10:45:58.983520 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:58.983436    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:45:59.117643 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:59.117611    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:45:59.258415 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:59.258380    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"]
Sep 09 10:45:59.259840 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:45:59.259817    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de794cb_81be_4600_b724_e4ff6193ae3b.slice/crio-7c94f0c056d97a64d45440f1f8f50b0cd65c9a68316dee5573099a66c377cd29 WatchSource:0}: Error finding container 7c94f0c056d97a64d45440f1f8f50b0cd65c9a68316dee5573099a66c377cd29: Status 404 returned error can't find the container with id 7c94f0c056d97a64d45440f1f8f50b0cd65c9a68316dee5573099a66c377cd29
Sep 09 10:45:59.262117 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:59.262080    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:59.262182 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:59.262150    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:59.262229 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:59.262185    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:59.659485 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:59.659386    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" event={"ID":"7de794cb-81be-4600-b724-e4ff6193ae3b","Type":"ContainerStarted","Data":"fcf865eeee8e4bc0efcc60d422eeac49a4f6a7d189a2b1ebbc8314b08fa6d5aa"}
Sep 09 10:45:59.659485 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:59.659443    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" event={"ID":"7de794cb-81be-4600-b724-e4ff6193ae3b","Type":"ContainerStarted","Data":"7c94f0c056d97a64d45440f1f8f50b0cd65c9a68316dee5573099a66c377cd29"}
Sep 09 10:45:59.683881 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:59.683834    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" podStartSLOduration=1.683815059 podStartE2EDuration="1.683815059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:59.680079509 +0000 UTC m=+1199.343718484" watchObservedRunningTime="2026-09-09 10:45:59.683815059 +0000 UTC m=+1199.347454035"
Sep 09 10:45:59.984613 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:45:59.984538    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:46:00.989095 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:00.989070    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:46:01.028163 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:01.028131    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:01.056628 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:01.056604    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:46:01.060330 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:01.060313    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:46:01.063283 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:01.063262    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:46:01.067154 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:01.067138    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:46:01.126074 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:01.126046    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:01.669358 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:01.669332    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:46:01.670307 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:01.670289    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:46:02.174743 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.174704    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"]
Sep 09 10:46:02.178585 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.178556    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.181394 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.181368    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dfb9af4b-serving-cert\""
Sep 09 10:46:02.191358 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.191332    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"]
Sep 09 10:46:02.226393 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.226356    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:02.237586 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.237550    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-openshift-service-ca\") pod \"maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.237586 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.237586    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-maas-api-tls\") pod \"maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.237816 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.237611    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-metrics-certs\") pod \"maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.237816 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.237714    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7mqn\" (UniqueName: \"kubernetes.io/projected/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-kube-api-access-h7mqn\") pod \"maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.324958 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.324925    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:02.338728 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.338687    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mqn\" (UniqueName: \"kubernetes.io/projected/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-kube-api-access-h7mqn\") pod \"maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.338925 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.338758    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-openshift-service-ca\") pod \"maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.338925 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.338808    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-maas-api-tls\") pod \"maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.338925 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.338842    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-metrics-certs\") pod \"maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.339105 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:46:02.338930    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-serving-cert: secret "maas-api-e2e-ait-dfb9af4b-serving-cert" not found
Sep 09 10:46:02.339105 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:46:02.339009    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-maas-api-tls podName:bd5869c8-0dc1-4ce9-b331-df5e19b1c37d nodeName:}" failed. No retries permitted until 2026-09-09 10:46:02.838989557 +0000 UTC m=+1202.502628531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-maas-api-tls") pod "maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf" (UID: "bd5869c8-0dc1-4ce9-b331-df5e19b1c37d") : secret "maas-api-e2e-ait-dfb9af4b-serving-cert" not found
Sep 09 10:46:02.339463 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.339441    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-openshift-service-ca\") pod \"maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.341344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.341328    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-metrics-certs\") pod \"maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.347126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.347108    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7mqn\" (UniqueName: \"kubernetes.io/projected/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-kube-api-access-h7mqn\") pod \"maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.774275 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.774245    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:02.847879 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.846935    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-maas-api-tls\") pod \"maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.854932 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.854899    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-maas-api-tls\") pod \"maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:02.869392 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:02.869363    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:03.091956 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.091926    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:03.216785 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.216748    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:46:03.252668 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.252644    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-credential-socket\") pod \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") "
Sep 09 10:46:03.252852 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.252682    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-podinfo\") pod \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") "
Sep 09 10:46:03.252852 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.252731    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-data\") pod \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") "
Sep 09 10:46:03.252852 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.252792    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8rtl\" (UniqueName: \"kubernetes.io/projected/e76597ba-f9ca-424e-8d2d-b5e3434fa026-kube-api-access-d8rtl\") pod \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") "
Sep 09 10:46:03.252852 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.252821    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-envoy\") pod \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") "
Sep 09 10:46:03.253050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.252860    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-workload-certs\") pod \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") "
Sep 09 10:46:03.253050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.252893    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istiod-ca-cert\") pod \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") "
Sep 09 10:46:03.253050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.252943    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-workload-socket\") pod \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") "
Sep 09 10:46:03.253050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.252971    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-token\") pod \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\" (UID: \"e76597ba-f9ca-424e-8d2d-b5e3434fa026\") "
Sep 09 10:46:03.253050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.252983    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e76597ba-f9ca-424e-8d2d-b5e3434fa026" (UID: "e76597ba-f9ca-424e-8d2d-b5e3434fa026"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:03.253310 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.253179    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e76597ba-f9ca-424e-8d2d-b5e3434fa026" (UID: "e76597ba-f9ca-424e-8d2d-b5e3434fa026"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:03.253310 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.253294    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e76597ba-f9ca-424e-8d2d-b5e3434fa026" (UID: "e76597ba-f9ca-424e-8d2d-b5e3434fa026"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:46:03.253475 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.253452    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-workload-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:03.253558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.253479    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istiod-ca-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:03.253558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.253492    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-credential-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:03.253558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.253534    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e76597ba-f9ca-424e-8d2d-b5e3434fa026" (UID: "e76597ba-f9ca-424e-8d2d-b5e3434fa026"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:03.253678 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.253624    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-data" (OuterVolumeSpecName: "istio-data") pod "e76597ba-f9ca-424e-8d2d-b5e3434fa026" (UID: "e76597ba-f9ca-424e-8d2d-b5e3434fa026"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:03.255569 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.255541    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-token" (OuterVolumeSpecName: "istio-token") pod "e76597ba-f9ca-424e-8d2d-b5e3434fa026" (UID: "e76597ba-f9ca-424e-8d2d-b5e3434fa026"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:46:03.255801 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.255778    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76597ba-f9ca-424e-8d2d-b5e3434fa026-kube-api-access-d8rtl" (OuterVolumeSpecName: "kube-api-access-d8rtl") pod "e76597ba-f9ca-424e-8d2d-b5e3434fa026" (UID: "e76597ba-f9ca-424e-8d2d-b5e3434fa026"). InnerVolumeSpecName "kube-api-access-d8rtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:46:03.255925 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.255889    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e76597ba-f9ca-424e-8d2d-b5e3434fa026" (UID: "e76597ba-f9ca-424e-8d2d-b5e3434fa026"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:03.256003 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.255984    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e76597ba-f9ca-424e-8d2d-b5e3434fa026" (UID: "e76597ba-f9ca-424e-8d2d-b5e3434fa026"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:46:03.354756 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.354723    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-workload-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:03.354896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.354760    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:03.354896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.354792    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-podinfo\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:03.354896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.354806    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-data\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:03.354896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.354820    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8rtl\" (UniqueName: \"kubernetes.io/projected/e76597ba-f9ca-424e-8d2d-b5e3434fa026-kube-api-access-d8rtl\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:03.354896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.354834    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e76597ba-f9ca-424e-8d2d-b5e3434fa026-istio-envoy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:03.386078 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.386055    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"]
Sep 09 10:46:03.388835 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:46:03.388809    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5869c8_0dc1_4ce9_b331_df5e19b1c37d.slice/crio-c0aeab8b375781e3d403c456dd8008045634f69234b781bd48fbcc8759ae3c42 WatchSource:0}: Error finding container c0aeab8b375781e3d403c456dd8008045634f69234b781bd48fbcc8759ae3c42: Status 404 returned error can't find the container with id c0aeab8b375781e3d403c456dd8008045634f69234b781bd48fbcc8759ae3c42
Sep 09 10:46:03.679590 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.679509    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf" event={"ID":"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d","Type":"ContainerStarted","Data":"c0aeab8b375781e3d403c456dd8008045634f69234b781bd48fbcc8759ae3c42"}
Sep 09 10:46:03.681216 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.681182    2570 generic.go:356] "Generic (PLEG): container finished" podID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerID="9279066163d134447d560077a25b6fc690741c0667b7103412ec966f2f87d263" exitCode=0
Sep 09 10:46:03.681367 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.681236    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"
Sep 09 10:46:03.681367 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.681263    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5" event={"ID":"e76597ba-f9ca-424e-8d2d-b5e3434fa026","Type":"ContainerDied","Data":"9279066163d134447d560077a25b6fc690741c0667b7103412ec966f2f87d263"}
Sep 09 10:46:03.681367 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.681320    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5" event={"ID":"e76597ba-f9ca-424e-8d2d-b5e3434fa026","Type":"ContainerDied","Data":"e2ab5682936ef81cf72aa4177e5f19097677c44003ebede9b55ace197da9147e"}
Sep 09 10:46:03.681367 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.681349    2570 scope.go:117] "RemoveContainer" containerID="9279066163d134447d560077a25b6fc690741c0667b7103412ec966f2f87d263"
Sep 09 10:46:03.690948 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.690928    2570 scope.go:117] "RemoveContainer" containerID="4622c5f62b5ae8e150fb098b2a0777776c405e7749bef0dc07a41f9c237234ed"
Sep 09 10:46:03.698876 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.698857    2570 scope.go:117] "RemoveContainer" containerID="9279066163d134447d560077a25b6fc690741c0667b7103412ec966f2f87d263"
Sep 09 10:46:03.699127 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:46:03.699107    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9279066163d134447d560077a25b6fc690741c0667b7103412ec966f2f87d263\": container with ID starting with 9279066163d134447d560077a25b6fc690741c0667b7103412ec966f2f87d263 not found: ID does not exist" containerID="9279066163d134447d560077a25b6fc690741c0667b7103412ec966f2f87d263"
Sep 09 10:46:03.699209 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.699138    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9279066163d134447d560077a25b6fc690741c0667b7103412ec966f2f87d263"} err="failed to get container status \"9279066163d134447d560077a25b6fc690741c0667b7103412ec966f2f87d263\": rpc error: code = NotFound desc = could not find container \"9279066163d134447d560077a25b6fc690741c0667b7103412ec966f2f87d263\": container with ID starting with 9279066163d134447d560077a25b6fc690741c0667b7103412ec966f2f87d263 not found: ID does not exist"
Sep 09 10:46:03.699209 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.699162    2570 scope.go:117] "RemoveContainer" containerID="4622c5f62b5ae8e150fb098b2a0777776c405e7749bef0dc07a41f9c237234ed"
Sep 09 10:46:03.699428 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:46:03.699411    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4622c5f62b5ae8e150fb098b2a0777776c405e7749bef0dc07a41f9c237234ed\": container with ID starting with 4622c5f62b5ae8e150fb098b2a0777776c405e7749bef0dc07a41f9c237234ed not found: ID does not exist" containerID="4622c5f62b5ae8e150fb098b2a0777776c405e7749bef0dc07a41f9c237234ed"
Sep 09 10:46:03.699468 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.699433    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4622c5f62b5ae8e150fb098b2a0777776c405e7749bef0dc07a41f9c237234ed"} err="failed to get container status \"4622c5f62b5ae8e150fb098b2a0777776c405e7749bef0dc07a41f9c237234ed\": rpc error: code = NotFound desc = could not find container \"4622c5f62b5ae8e150fb098b2a0777776c405e7749bef0dc07a41f9c237234ed\": container with ID starting with 4622c5f62b5ae8e150fb098b2a0777776c405e7749bef0dc07a41f9c237234ed not found: ID does not exist"
Sep 09 10:46:03.706991 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.706972    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"]
Sep 09 10:46:03.714900 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:03.714882    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-92ac8218-gw-openshift-default-7dddd7768-rvqj5"]
Sep 09 10:46:04.688229 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:04.688191    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf" event={"ID":"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d","Type":"ContainerStarted","Data":"f9a6f1f8738e01917250f272a3ab554d133df9c45f99cb1b3052181a5b810dc6"}
Sep 09 10:46:04.688673 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:04.688324    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:04.706754 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:04.706705    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf" podStartSLOduration=2.171242306 podStartE2EDuration="2.706689171s" totalImagesPullingTime="535.446865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:46:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:46:04.706072881 +0000 UTC m=+1204.369711854" watchObservedRunningTime="2026-09-09 10:46:04.706689171 +0000 UTC m=+1204.370328144"
Sep 09 10:46:04.839344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:04.839313    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" path="/var/lib/kubelet/pods/e76597ba-f9ca-424e-8d2d-b5e3434fa026/volumes"
Sep 09 10:46:10.699020 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:10.698991    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:24.581253 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:24.581218    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:25.814707 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:25.814673    2570 generic.go:356] "Generic (PLEG): container finished" podID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerID="fcf865eeee8e4bc0efcc60d422eeac49a4f6a7d189a2b1ebbc8314b08fa6d5aa" exitCode=0
Sep 09 10:46:25.815144 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:25.814748    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" event={"ID":"7de794cb-81be-4600-b724-e4ff6193ae3b","Type":"ContainerDied","Data":"fcf865eeee8e4bc0efcc60d422eeac49a4f6a7d189a2b1ebbc8314b08fa6d5aa"}
Sep 09 10:46:25.815229 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:25.815215    2570 scope.go:117] "RemoveContainer" containerID="fcf865eeee8e4bc0efcc60d422eeac49a4f6a7d189a2b1ebbc8314b08fa6d5aa"
Sep 09 10:46:25.816396 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:25.816364    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:46:25.816490 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:25.816445    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:46:25.816490 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:25.816483    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:46:25.974096 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:25.974000    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:26.823173 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:26.823140    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" event={"ID":"7de794cb-81be-4600-b724-e4ff6193ae3b","Type":"ContainerStarted","Data":"c9c63a447b8c052116e1b863a37489b5aa97ae7e20389fbda0d9b521ee4ab1cd"}
Sep 09 10:46:26.984082 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:26.984043    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:46:26.985476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:26.985446    2570 patch_prober.go:28] interesting pod/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 09 10:46:26.985735 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:26.985687    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 09 10:46:27.983883 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:27.983842    2570 patch_prober.go:28] interesting pod/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 09 10:46:27.984299 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:27.983903    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 09 10:46:28.597929 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.597887    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"]
Sep 09 10:46:28.598319 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.598307    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerName="istio-proxy"
Sep 09 10:46:28.598379 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.598321    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerName="istio-proxy"
Sep 09 10:46:28.598379 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.598337    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerName="istio-proxy"
Sep 09 10:46:28.598379 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.598343    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerName="istio-proxy"
Sep 09 10:46:28.598486 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.598408    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerName="istio-proxy"
Sep 09 10:46:28.598486 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.598416    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e76597ba-f9ca-424e-8d2d-b5e3434fa026" containerName="istio-proxy"
Sep 09 10:46:28.600400 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.600383    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.604224 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.604202    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-40f1e9\"/\"default-dockercfg-7bdnx\""
Sep 09 10:46:28.604348 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.604232    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-40f1e9\"/\"openshift-service-ca.crt\""
Sep 09 10:46:28.604348 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.604259    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-40f1e9\"/\"kube-root-ca.crt\""
Sep 09 10:46:28.604348 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.604237    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-40f1e9\"/\"test-model-40f1e9-kserve-self-signed-certs\""
Sep 09 10:46:28.612036 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.612011    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"]
Sep 09 10:46:28.624331 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.624295    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:28.716795 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.716729    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-model-cache\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.716795 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.716794    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-tls-certs\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.717023 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.716826    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-tmp-dir\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.717023 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.716889    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flzqh\" (UniqueName: \"kubernetes.io/projected/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-kube-api-access-flzqh\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.717023 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.716923    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-home\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.717023 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.716995    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-dshm\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.817794 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.817729    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flzqh\" (UniqueName: \"kubernetes.io/projected/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-kube-api-access-flzqh\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.817989 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.817829    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-home\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.817989 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.817891    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-dshm\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.817989 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.817916    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-model-cache\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.817989 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.817934    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-tls-certs\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.817989 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.817952    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-tmp-dir\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.818373 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.818352    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-home\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.818469 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.818377    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-model-cache\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.818469 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.818423    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-tmp-dir\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.820164 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.820144    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-dshm\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.820561 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.820543    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-tls-certs\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.827493 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.827459    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flzqh\" (UniqueName: \"kubernetes.io/projected/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-kube-api-access-flzqh\") pod \"test-model-40f1e9-kserve-644b64db75-nqss2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") " pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.910167 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.910074    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:28.984324 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.984249    2570 patch_prober.go:28] interesting pod/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 09 10:46:28.984650 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:28.984320    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 09 10:46:29.203484 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:29.203460    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"]
Sep 09 10:46:29.204918 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:46:29.204889    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14f5f1df_cd2f_4c17_b439_ecd6b1915da2.slice/crio-ae3fdb3c40b310bca510fbf4d1b364b54eac85f8abd765d80cdc764bb529c911 WatchSource:0}: Error finding container ae3fdb3c40b310bca510fbf4d1b364b54eac85f8abd765d80cdc764bb529c911: Status 404 returned error can't find the container with id ae3fdb3c40b310bca510fbf4d1b364b54eac85f8abd765d80cdc764bb529c911
Sep 09 10:46:29.839727 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:29.839690    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2" event={"ID":"14f5f1df-cd2f-4c17-b439-ecd6b1915da2","Type":"ContainerStarted","Data":"3b1ad11d78411dd37aed74e66f96ab4bcd747593ee950a324dc1b4e6a9829ae3"}
Sep 09 10:46:29.839966 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:29.839739    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2" event={"ID":"14f5f1df-cd2f-4c17-b439-ecd6b1915da2","Type":"ContainerStarted","Data":"ae3fdb3c40b310bca510fbf4d1b364b54eac85f8abd765d80cdc764bb529c911"}
Sep 09 10:46:29.862684 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:29.862609    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2" podStartSLOduration=1.862582308 podStartE2EDuration="1.862582308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:46:29.85922064 +0000 UTC m=+1229.522859615" watchObservedRunningTime="2026-09-09 10:46:29.862582308 +0000 UTC m=+1229.526221283"
Sep 09 10:46:29.983970 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:29.983932    2570 patch_prober.go:28] interesting pod/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 09 10:46:29.984147 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:29.983997    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 09 10:46:30.636254 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:30.636223    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:30.984530 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:30.984438    2570 patch_prober.go:28] interesting pod/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 09 10:46:30.984530 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:30.984502    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 09 10:46:31.669869 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:31.669831    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:46:31.984958 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:31.984845    2570 patch_prober.go:28] interesting pod/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 09 10:46:31.984958 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:31.984918    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 09 10:46:32.089935 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:32.089903    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:32.988511 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:32.988478    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:46:32.989570 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:32.989554    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:46:33.042355 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.042318    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"]
Sep 09 10:46:33.042581 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.042556    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf" podUID="bd5869c8-0dc1-4ce9-b331-df5e19b1c37d" containerName="maas-api" containerID="cri-o://f9a6f1f8738e01917250f272a3ab554d133df9c45f99cb1b3052181a5b810dc6" gracePeriod=30
Sep 09 10:46:33.074682 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.074640    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:33.196452 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.196409    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:33.293747 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.293717    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:33.301415 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.301392    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:33.367330 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.367298    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:33.376804 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.376758    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-metrics-certs\") pod \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") "
Sep 09 10:46:33.376884 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.376848    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7mqn\" (UniqueName: \"kubernetes.io/projected/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-kube-api-access-h7mqn\") pod \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") "
Sep 09 10:46:33.376927 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.376906    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-maas-api-tls\") pod \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") "
Sep 09 10:46:33.376983 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.376968    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-openshift-service-ca\") pod \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\" (UID: \"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d\") "
Sep 09 10:46:33.377426 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.377393    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bd5869c8-0dc1-4ce9-b331-df5e19b1c37d" (UID: "bd5869c8-0dc1-4ce9-b331-df5e19b1c37d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:46:33.379049 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.379017    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bd5869c8-0dc1-4ce9-b331-df5e19b1c37d" (UID: "bd5869c8-0dc1-4ce9-b331-df5e19b1c37d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:46:33.379151 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.379070    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bd5869c8-0dc1-4ce9-b331-df5e19b1c37d" (UID: "bd5869c8-0dc1-4ce9-b331-df5e19b1c37d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:46:33.379151 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.379077    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-kube-api-access-h7mqn" (OuterVolumeSpecName: "kube-api-access-h7mqn") pod "bd5869c8-0dc1-4ce9-b331-df5e19b1c37d" (UID: "bd5869c8-0dc1-4ce9-b331-df5e19b1c37d"). InnerVolumeSpecName "kube-api-access-h7mqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:46:33.478284 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.478244    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:33.478284 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.478284    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7mqn\" (UniqueName: \"kubernetes.io/projected/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-kube-api-access-h7mqn\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:33.478602 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.478298    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-maas-api-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:33.478602 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.478312    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d-openshift-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:33.869692 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.869657    2570 generic.go:356] "Generic (PLEG): container finished" podID="bd5869c8-0dc1-4ce9-b331-df5e19b1c37d" containerID="f9a6f1f8738e01917250f272a3ab554d133df9c45f99cb1b3052181a5b810dc6" exitCode=0
Sep 09 10:46:33.869692 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.869695    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"
Sep 09 10:46:33.869939 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.869756    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf" event={"ID":"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d","Type":"ContainerDied","Data":"f9a6f1f8738e01917250f272a3ab554d133df9c45f99cb1b3052181a5b810dc6"}
Sep 09 10:46:33.869939 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.869838    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf" event={"ID":"bd5869c8-0dc1-4ce9-b331-df5e19b1c37d","Type":"ContainerDied","Data":"c0aeab8b375781e3d403c456dd8008045634f69234b781bd48fbcc8759ae3c42"}
Sep 09 10:46:33.869939 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.869861    2570 scope.go:117] "RemoveContainer" containerID="f9a6f1f8738e01917250f272a3ab554d133df9c45f99cb1b3052181a5b810dc6"
Sep 09 10:46:33.879512 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.879493    2570 scope.go:117] "RemoveContainer" containerID="f9a6f1f8738e01917250f272a3ab554d133df9c45f99cb1b3052181a5b810dc6"
Sep 09 10:46:33.879818 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:46:33.879794    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a6f1f8738e01917250f272a3ab554d133df9c45f99cb1b3052181a5b810dc6\": container with ID starting with f9a6f1f8738e01917250f272a3ab554d133df9c45f99cb1b3052181a5b810dc6 not found: ID does not exist" containerID="f9a6f1f8738e01917250f272a3ab554d133df9c45f99cb1b3052181a5b810dc6"
Sep 09 10:46:33.879877 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.879831    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a6f1f8738e01917250f272a3ab554d133df9c45f99cb1b3052181a5b810dc6"} err="failed to get container status \"f9a6f1f8738e01917250f272a3ab554d133df9c45f99cb1b3052181a5b810dc6\": rpc error: code = NotFound desc = could not find container \"f9a6f1f8738e01917250f272a3ab554d133df9c45f99cb1b3052181a5b810dc6\": container with ID starting with f9a6f1f8738e01917250f272a3ab554d133df9c45f99cb1b3052181a5b810dc6 not found: ID does not exist"
Sep 09 10:46:33.894252 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.894225    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"]
Sep 09 10:46:33.900212 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:33.900188    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dfb9af4b-6679cc6bd9-n5rtf"]
Sep 09 10:46:34.278133 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:34.278104    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:34.536515 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:34.536419    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"]
Sep 09 10:46:34.556026 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:34.555993    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:34.840125 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:34.840037    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd5869c8-0dc1-4ce9-b331-df5e19b1c37d" path="/var/lib/kubelet/pods/bd5869c8-0dc1-4ce9-b331-df5e19b1c37d/volumes"
Sep 09 10:46:34.876049 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:34.876011    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerName="istio-proxy" containerID="cri-o://c9c63a447b8c052116e1b863a37489b5aa97ae7e20389fbda0d9b521ee4ab1cd" gracePeriod=30
Sep 09 10:46:34.981123 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:34.981088    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:34.996552 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:34.996518    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"]
Sep 09 10:46:34.997344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:34.996833    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264" podUID="328ffd1f-d294-4a7f-a245-3b280a9e8400" containerName="maas-api" containerID="cri-o://71cc44152c6f793cafa7b15fa64501dec7500f7ff0156904b6ff0add1adebb4f" gracePeriod=30
Sep 09 10:46:35.063940 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.063908    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"]
Sep 09 10:46:35.064196 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.064154    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6" podUID="eb70e500-9155-46ab-b392-560019ad9384" containerName="payload-processing" containerID="cri-o://3a297d23753592b023658d906a76335522d6fbd6dd8db2670391bf9db56bfe72" gracePeriod=30
Sep 09 10:46:35.066283 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.066257    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:35.140189 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.140100    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:35.246593 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.246559    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:35.294467 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.294439    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"]
Sep 09 10:46:35.294954 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.294940    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd5869c8-0dc1-4ce9-b331-df5e19b1c37d" containerName="maas-api"
Sep 09 10:46:35.295018 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.294958    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd5869c8-0dc1-4ce9-b331-df5e19b1c37d" containerName="maas-api"
Sep 09 10:46:35.295122 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.295106    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd5869c8-0dc1-4ce9-b331-df5e19b1c37d" containerName="maas-api"
Sep 09 10:46:35.297550 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.297529    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.300996 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.300973    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d84c5166-openshift-default-dockercfg-nqzsx\""
Sep 09 10:46:35.318607 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.318579    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"]
Sep 09 10:46:35.336119 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.336095    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:46:35.341724 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.341695    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:35.359446 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.359424    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.404449    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-metrics-certs\") pod \"328ffd1f-d294-4a7f-a245-3b280a9e8400\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") "
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.404522    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/328ffd1f-d294-4a7f-a245-3b280a9e8400-openshift-service-ca\") pod \"328ffd1f-d294-4a7f-a245-3b280a9e8400\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") "
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.404959    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328ffd1f-d294-4a7f-a245-3b280a9e8400-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "328ffd1f-d294-4a7f-a245-3b280a9e8400" (UID: "328ffd1f-d294-4a7f-a245-3b280a9e8400"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.405099    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb70e500-9155-46ab-b392-560019ad9384-plugins-config-volume\") pod \"eb70e500-9155-46ab-b392-560019ad9384\" (UID: \"eb70e500-9155-46ab-b392-560019ad9384\") "
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.405169    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz9fh\" (UniqueName: \"kubernetes.io/projected/eb70e500-9155-46ab-b392-560019ad9384-kube-api-access-gz9fh\") pod \"eb70e500-9155-46ab-b392-560019ad9384\" (UID: \"eb70e500-9155-46ab-b392-560019ad9384\") "
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.405246    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-maas-api-tls\") pod \"328ffd1f-d294-4a7f-a245-3b280a9e8400\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") "
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.405273    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqc42\" (UniqueName: \"kubernetes.io/projected/328ffd1f-d294-4a7f-a245-3b280a9e8400-kube-api-access-fqc42\") pod \"328ffd1f-d294-4a7f-a245-3b280a9e8400\" (UID: \"328ffd1f-d294-4a7f-a245-3b280a9e8400\") "
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.405493    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-envoy\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.405562    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-data\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.405589    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-token\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.405641    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-workload-socket\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.405735    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsrvx\" (UniqueName: \"kubernetes.io/projected/e07842b7-679c-4843-8b00-9aa214eefd5c-kube-api-access-hsrvx\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.405817    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-workload-certs\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.405850    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-credential-socket\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.408719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.405888    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-podinfo\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.409649 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.405913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07842b7-679c-4843-8b00-9aa214eefd5c-istiod-ca-cert\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.409649 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.406024    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/328ffd1f-d294-4a7f-a245-3b280a9e8400-openshift-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:35.415018 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.412144    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328ffd1f-d294-4a7f-a245-3b280a9e8400-kube-api-access-fqc42" (OuterVolumeSpecName: "kube-api-access-fqc42") pod "328ffd1f-d294-4a7f-a245-3b280a9e8400" (UID: "328ffd1f-d294-4a7f-a245-3b280a9e8400"). InnerVolumeSpecName "kube-api-access-fqc42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:46:35.415179 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.415056    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb70e500-9155-46ab-b392-560019ad9384-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb70e500-9155-46ab-b392-560019ad9384" (UID: "eb70e500-9155-46ab-b392-560019ad9384"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:46:35.415661 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.415558    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb70e500-9155-46ab-b392-560019ad9384-kube-api-access-gz9fh" (OuterVolumeSpecName: "kube-api-access-gz9fh") pod "eb70e500-9155-46ab-b392-560019ad9384" (UID: "eb70e500-9155-46ab-b392-560019ad9384"). InnerVolumeSpecName "kube-api-access-gz9fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:46:35.418299 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.418230    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "328ffd1f-d294-4a7f-a245-3b280a9e8400" (UID: "328ffd1f-d294-4a7f-a245-3b280a9e8400"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:46:35.419402 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.419245    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "328ffd1f-d294-4a7f-a245-3b280a9e8400" (UID: "328ffd1f-d294-4a7f-a245-3b280a9e8400"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:46:35.474207 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.474174    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:35.507510 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.507473    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-envoy\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.507673 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.507523    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-data\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.507673 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.507635    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-token\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.507786 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.507698    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-workload-socket\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.507847 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.507817    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrvx\" (UniqueName: \"kubernetes.io/projected/e07842b7-679c-4843-8b00-9aa214eefd5c-kube-api-access-hsrvx\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.507903 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.507871    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-workload-certs\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.507960 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.507894    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-data\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.507960 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.507902    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-credential-socket\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.508106 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.507982    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-workload-socket\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.508106 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.507985    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-podinfo\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.508106 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.508017    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07842b7-679c-4843-8b00-9aa214eefd5c-istiod-ca-cert\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.508261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.508119    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-credential-socket\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.508261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.508144    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb70e500-9155-46ab-b392-560019ad9384-plugins-config-volume\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:35.508261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.508162    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz9fh\" (UniqueName: \"kubernetes.io/projected/eb70e500-9155-46ab-b392-560019ad9384-kube-api-access-gz9fh\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:35.508261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.508177    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-maas-api-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:35.508261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.508183    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-workload-certs\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.508261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.508191    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqc42\" (UniqueName: \"kubernetes.io/projected/328ffd1f-d294-4a7f-a245-3b280a9e8400-kube-api-access-fqc42\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:35.508261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.508229    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/328ffd1f-d294-4a7f-a245-3b280a9e8400-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:35.508801 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.508753    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07842b7-679c-4843-8b00-9aa214eefd5c-istiod-ca-cert\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.509978 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.509958    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-envoy\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.510294 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.510276    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-podinfo\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.517919 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.517894    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrvx\" (UniqueName: \"kubernetes.io/projected/e07842b7-679c-4843-8b00-9aa214eefd5c-kube-api-access-hsrvx\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.518053 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.518035    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-token\") pod \"e2e-derive-d84c5166-openshift-default-747db49656-dntb6\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") " pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.568694 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.568637    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:35.633455 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.633423    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:35.725050 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.725023    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:35.882944 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.882912    2570 generic.go:356] "Generic (PLEG): container finished" podID="eb70e500-9155-46ab-b392-560019ad9384" containerID="3a297d23753592b023658d906a76335522d6fbd6dd8db2670391bf9db56bfe72" exitCode=0
Sep 09 10:46:35.883144 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.882973    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"
Sep 09 10:46:35.883144 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.882987    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6" event={"ID":"eb70e500-9155-46ab-b392-560019ad9384","Type":"ContainerDied","Data":"3a297d23753592b023658d906a76335522d6fbd6dd8db2670391bf9db56bfe72"}
Sep 09 10:46:35.883144 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.883041    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6" event={"ID":"eb70e500-9155-46ab-b392-560019ad9384","Type":"ContainerDied","Data":"e7ce11f12db651250583bd1142624a4caf5b2b4b3e7a46a7c4ba93af94e3a306"}
Sep 09 10:46:35.883144 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.883091    2570 scope.go:117] "RemoveContainer" containerID="3a297d23753592b023658d906a76335522d6fbd6dd8db2670391bf9db56bfe72"
Sep 09 10:46:35.884705 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.884522    2570 generic.go:356] "Generic (PLEG): container finished" podID="328ffd1f-d294-4a7f-a245-3b280a9e8400" containerID="71cc44152c6f793cafa7b15fa64501dec7500f7ff0156904b6ff0add1adebb4f" exitCode=0
Sep 09 10:46:35.884705 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.884560    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"
Sep 09 10:46:35.884705 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.884587    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264" event={"ID":"328ffd1f-d294-4a7f-a245-3b280a9e8400","Type":"ContainerDied","Data":"71cc44152c6f793cafa7b15fa64501dec7500f7ff0156904b6ff0add1adebb4f"}
Sep 09 10:46:35.884705 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.884620    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264" event={"ID":"328ffd1f-d294-4a7f-a245-3b280a9e8400","Type":"ContainerDied","Data":"a12f5bedb8bff9c5b37d0fdc8e17d25c675cfdac4b88232f9012c6b3ac118a4c"}
Sep 09 10:46:35.893621 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.893603    2570 scope.go:117] "RemoveContainer" containerID="3a297d23753592b023658d906a76335522d6fbd6dd8db2670391bf9db56bfe72"
Sep 09 10:46:35.894212 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:46:35.894186    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a297d23753592b023658d906a76335522d6fbd6dd8db2670391bf9db56bfe72\": container with ID starting with 3a297d23753592b023658d906a76335522d6fbd6dd8db2670391bf9db56bfe72 not found: ID does not exist" containerID="3a297d23753592b023658d906a76335522d6fbd6dd8db2670391bf9db56bfe72"
Sep 09 10:46:35.894340 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.894222    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a297d23753592b023658d906a76335522d6fbd6dd8db2670391bf9db56bfe72"} err="failed to get container status \"3a297d23753592b023658d906a76335522d6fbd6dd8db2670391bf9db56bfe72\": rpc error: code = NotFound desc = could not find container \"3a297d23753592b023658d906a76335522d6fbd6dd8db2670391bf9db56bfe72\": container with ID starting with 3a297d23753592b023658d906a76335522d6fbd6dd8db2670391bf9db56bfe72 not found: ID does not exist"
Sep 09 10:46:35.894340 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.894246    2570 scope.go:117] "RemoveContainer" containerID="71cc44152c6f793cafa7b15fa64501dec7500f7ff0156904b6ff0add1adebb4f"
Sep 09 10:46:35.902954 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.902934    2570 scope.go:117] "RemoveContainer" containerID="71cc44152c6f793cafa7b15fa64501dec7500f7ff0156904b6ff0add1adebb4f"
Sep 09 10:46:35.903246 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:46:35.903229    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71cc44152c6f793cafa7b15fa64501dec7500f7ff0156904b6ff0add1adebb4f\": container with ID starting with 71cc44152c6f793cafa7b15fa64501dec7500f7ff0156904b6ff0add1adebb4f not found: ID does not exist" containerID="71cc44152c6f793cafa7b15fa64501dec7500f7ff0156904b6ff0add1adebb4f"
Sep 09 10:46:35.903324 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.903252    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cc44152c6f793cafa7b15fa64501dec7500f7ff0156904b6ff0add1adebb4f"} err="failed to get container status \"71cc44152c6f793cafa7b15fa64501dec7500f7ff0156904b6ff0add1adebb4f\": rpc error: code = NotFound desc = could not find container \"71cc44152c6f793cafa7b15fa64501dec7500f7ff0156904b6ff0add1adebb4f\": container with ID starting with 71cc44152c6f793cafa7b15fa64501dec7500f7ff0156904b6ff0add1adebb4f not found: ID does not exist"
Sep 09 10:46:35.913089 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.913062    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"]
Sep 09 10:46:35.921266 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.921225    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"]
Sep 09 10:46:35.922977 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:46:35.922956    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07842b7_679c_4843_8b00_9aa214eefd5c.slice/crio-34de440a1b46a3b115768108cee59e0c6e32b4c53df9fbc29f176d5f1c36c0a4 WatchSource:0}: Error finding container 34de440a1b46a3b115768108cee59e0c6e32b4c53df9fbc29f176d5f1c36c0a4: Status 404 returned error can't find the container with id 34de440a1b46a3b115768108cee59e0c6e32b4c53df9fbc29f176d5f1c36c0a4
Sep 09 10:46:35.925304 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.925242    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4bbfe3-5c84fdcbfc-8tkg6"]
Sep 09 10:46:35.925600 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.925571    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:46:35.925681 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.925665    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:46:35.925729 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.925713    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:46:35.936378 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.936325    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"]
Sep 09 10:46:35.946305 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:35.946282    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4bbfe3-c8d646bf6-qc264"]
Sep 09 10:46:36.844170 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:36.844119    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="328ffd1f-d294-4a7f-a245-3b280a9e8400" path="/var/lib/kubelet/pods/328ffd1f-d294-4a7f-a245-3b280a9e8400/volumes"
Sep 09 10:46:36.844781 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:36.844742    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb70e500-9155-46ab-b392-560019ad9384" path="/var/lib/kubelet/pods/eb70e500-9155-46ab-b392-560019ad9384/volumes"
Sep 09 10:46:36.894656 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:36.894616    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6" event={"ID":"e07842b7-679c-4843-8b00-9aa214eefd5c","Type":"ContainerStarted","Data":"0c72618f07734104e4579752da96cabdabf39a00df5fe443927e47f7d04f8215"}
Sep 09 10:46:36.894904 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:36.894707    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6" event={"ID":"e07842b7-679c-4843-8b00-9aa214eefd5c","Type":"ContainerStarted","Data":"34de440a1b46a3b115768108cee59e0c6e32b4c53df9fbc29f176d5f1c36c0a4"}
Sep 09 10:46:36.922219 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:36.922162    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6" podStartSLOduration=1.9221444349999999 podStartE2EDuration="1.922144435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:46:36.920492846 +0000 UTC m=+1236.584131820" watchObservedRunningTime="2026-09-09 10:46:36.922144435 +0000 UTC m=+1236.585783409"
Sep 09 10:46:37.634601 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:37.634569    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:37.640031 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:37.640005    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:37.902237 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:37.902149    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:37.903706 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:37.903682    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:38.290485 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:38.290441    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:38.387127 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:38.387086    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:38.910813 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:38.910783    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:38.911223 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:38.910841    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:38.923546 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:38.923521    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:39.415667 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.415632    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"]
Sep 09 10:46:39.416339 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.416315    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb70e500-9155-46ab-b392-560019ad9384" containerName="payload-processing"
Sep 09 10:46:39.416403 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.416344    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb70e500-9155-46ab-b392-560019ad9384" containerName="payload-processing"
Sep 09 10:46:39.416403 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.416359    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328ffd1f-d294-4a7f-a245-3b280a9e8400" containerName="maas-api"
Sep 09 10:46:39.416403 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.416368    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="328ffd1f-d294-4a7f-a245-3b280a9e8400" containerName="maas-api"
Sep 09 10:46:39.416508 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.416480    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb70e500-9155-46ab-b392-560019ad9384" containerName="payload-processing"
Sep 09 10:46:39.416508 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.416493    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="328ffd1f-d294-4a7f-a245-3b280a9e8400" containerName="maas-api"
Sep 09 10:46:39.419468 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.419449    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.422256 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.422236    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d84c5166-serving-cert\""
Sep 09 10:46:39.430725 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.430659    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"]
Sep 09 10:46:39.460009 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.459978    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:39.549108 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.549075    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc581664-0839-46c2-9607-05d064e65ca1-maas-api-tls\") pod \"maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.549108 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.549112    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc581664-0839-46c2-9607-05d064e65ca1-openshift-service-ca\") pod \"maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.549339 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.549210    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc581664-0839-46c2-9607-05d064e65ca1-metrics-certs\") pod \"maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.549339 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.549294    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5g4q\" (UniqueName: \"kubernetes.io/projected/cc581664-0839-46c2-9607-05d064e65ca1-kube-api-access-s5g4q\") pod \"maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.563864 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.563832    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:39.649844 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.649803    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5g4q\" (UniqueName: \"kubernetes.io/projected/cc581664-0839-46c2-9607-05d064e65ca1-kube-api-access-s5g4q\") pod \"maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.650052 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.649879    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc581664-0839-46c2-9607-05d064e65ca1-maas-api-tls\") pod \"maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.650052 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.649904    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc581664-0839-46c2-9607-05d064e65ca1-openshift-service-ca\") pod \"maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.650052 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.649944    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc581664-0839-46c2-9607-05d064e65ca1-metrics-certs\") pod \"maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.650694 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.650661    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc581664-0839-46c2-9607-05d064e65ca1-openshift-service-ca\") pod \"maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.652683 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.652656    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc581664-0839-46c2-9607-05d064e65ca1-maas-api-tls\") pod \"maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.652838 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.652734    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc581664-0839-46c2-9607-05d064e65ca1-metrics-certs\") pod \"maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.660512 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.660447    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5g4q\" (UniqueName: \"kubernetes.io/projected/cc581664-0839-46c2-9607-05d064e65ca1-kube-api-access-s5g4q\") pod \"maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.661368 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.661335    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:39.734152 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.734051    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:39.924184 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:39.924157    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:46:40.000408 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.000372    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:40.025039 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.024999    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"]
Sep 09 10:46:40.025228 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:46:40.025191    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc581664_0839_46c2_9607_05d064e65ca1.slice/crio-e2935492d67af9d6a94b3555266ef2643a2c55d1d3beeb6a7e9ff9e61e51ca80 WatchSource:0}: Error finding container e2935492d67af9d6a94b3555266ef2643a2c55d1d3beeb6a7e9ff9e61e51ca80: Status 404 returned error can't find the container with id e2935492d67af9d6a94b3555266ef2643a2c55d1d3beeb6a7e9ff9e61e51ca80
Sep 09 10:46:40.110698 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.110665    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:40.131737 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.131713    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:46:40.190030 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.190001    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"]
Sep 09 10:46:40.190404 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.190392    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerName="istio-proxy"
Sep 09 10:46:40.190459 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.190406    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerName="istio-proxy"
Sep 09 10:46:40.190459 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.190417    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerName="istio-proxy"
Sep 09 10:46:40.190459 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.190423    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerName="istio-proxy"
Sep 09 10:46:40.190558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.190480    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerName="istio-proxy"
Sep 09 10:46:40.190558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.190490    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerName="istio-proxy"
Sep 09 10:46:40.192447 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.192432    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"
Sep 09 10:46:40.195227 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.195206    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d84c5166\""
Sep 09 10:46:40.195387 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.195311    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d84c5166-dockercfg-jj29k\""
Sep 09 10:46:40.202440 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.202416    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"]
Sep 09 10:46:40.256710 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.256620    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-token\") pod \"7de794cb-81be-4600-b724-e4ff6193ae3b\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") "
Sep 09 10:46:40.256710 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.256696    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-podinfo\") pod \"7de794cb-81be-4600-b724-e4ff6193ae3b\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") "
Sep 09 10:46:40.256908 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.256740    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-workload-certs\") pod \"7de794cb-81be-4600-b724-e4ff6193ae3b\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") "
Sep 09 10:46:40.256908 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.256820    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de794cb-81be-4600-b724-e4ff6193ae3b-istiod-ca-cert\") pod \"7de794cb-81be-4600-b724-e4ff6193ae3b\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") "
Sep 09 10:46:40.256908 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.256848    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-credential-socket\") pod \"7de794cb-81be-4600-b724-e4ff6193ae3b\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") "
Sep 09 10:46:40.256908 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.256874    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-envoy\") pod \"7de794cb-81be-4600-b724-e4ff6193ae3b\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") "
Sep 09 10:46:40.257121 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.256918    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tqlt\" (UniqueName: \"kubernetes.io/projected/7de794cb-81be-4600-b724-e4ff6193ae3b-kube-api-access-5tqlt\") pod \"7de794cb-81be-4600-b724-e4ff6193ae3b\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") "
Sep 09 10:46:40.257121 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.256970    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-data\") pod \"7de794cb-81be-4600-b724-e4ff6193ae3b\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") "
Sep 09 10:46:40.257121 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.257008    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-workload-socket\") pod \"7de794cb-81be-4600-b724-e4ff6193ae3b\" (UID: \"7de794cb-81be-4600-b724-e4ff6193ae3b\") "
Sep 09 10:46:40.257121 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.257047    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7de794cb-81be-4600-b724-e4ff6193ae3b" (UID: "7de794cb-81be-4600-b724-e4ff6193ae3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:40.257336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.257193    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dmmk\" (UniqueName: \"kubernetes.io/projected/48167dc3-0449-466b-8336-bb4c5a30f725-kube-api-access-6dmmk\") pod \"payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2\" (UID: \"48167dc3-0449-466b-8336-bb4c5a30f725\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"
Sep 09 10:46:40.257336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.257244    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-data" (OuterVolumeSpecName: "istio-data") pod "7de794cb-81be-4600-b724-e4ff6193ae3b" (UID: "7de794cb-81be-4600-b724-e4ff6193ae3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:40.257336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.257271    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48167dc3-0449-466b-8336-bb4c5a30f725-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2\" (UID: \"48167dc3-0449-466b-8336-bb4c5a30f725\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"
Sep 09 10:46:40.257336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.257305    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de794cb-81be-4600-b724-e4ff6193ae3b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7de794cb-81be-4600-b724-e4ff6193ae3b" (UID: "7de794cb-81be-4600-b724-e4ff6193ae3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:46:40.257661 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.257524    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de794cb-81be-4600-b724-e4ff6193ae3b-istiod-ca-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:40.257661 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.257546    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-data\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:40.257661 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.257560    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-workload-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:40.258090 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.258048    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7de794cb-81be-4600-b724-e4ff6193ae3b" (UID: "7de794cb-81be-4600-b724-e4ff6193ae3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:40.258333 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.258304    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7de794cb-81be-4600-b724-e4ff6193ae3b" (UID: "7de794cb-81be-4600-b724-e4ff6193ae3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:40.259589 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.259555    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de794cb-81be-4600-b724-e4ff6193ae3b-kube-api-access-5tqlt" (OuterVolumeSpecName: "kube-api-access-5tqlt") pod "7de794cb-81be-4600-b724-e4ff6193ae3b" (UID: "7de794cb-81be-4600-b724-e4ff6193ae3b"). InnerVolumeSpecName "kube-api-access-5tqlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:46:40.259657 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.259598    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "7de794cb-81be-4600-b724-e4ff6193ae3b" (UID: "7de794cb-81be-4600-b724-e4ff6193ae3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:46:40.259703 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.259673    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7de794cb-81be-4600-b724-e4ff6193ae3b" (UID: "7de794cb-81be-4600-b724-e4ff6193ae3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:46:40.260038 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.260017    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7de794cb-81be-4600-b724-e4ff6193ae3b" (UID: "7de794cb-81be-4600-b724-e4ff6193ae3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:40.358907 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.358859    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dmmk\" (UniqueName: \"kubernetes.io/projected/48167dc3-0449-466b-8336-bb4c5a30f725-kube-api-access-6dmmk\") pod \"payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2\" (UID: \"48167dc3-0449-466b-8336-bb4c5a30f725\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"
Sep 09 10:46:40.359112 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.358950    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48167dc3-0449-466b-8336-bb4c5a30f725-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2\" (UID: \"48167dc3-0449-466b-8336-bb4c5a30f725\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"
Sep 09 10:46:40.359178 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.359139    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-podinfo\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:40.359178 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.359157    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-credential-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:40.359178 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.359171    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-envoy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:40.359335 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.359184    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tqlt\" (UniqueName: \"kubernetes.io/projected/7de794cb-81be-4600-b724-e4ff6193ae3b-kube-api-access-5tqlt\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:40.359335 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.359197    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de794cb-81be-4600-b724-e4ff6193ae3b-workload-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:40.359335 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.359209    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de794cb-81be-4600-b724-e4ff6193ae3b-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:46:40.359711 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.359689    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48167dc3-0449-466b-8336-bb4c5a30f725-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2\" (UID: \"48167dc3-0449-466b-8336-bb4c5a30f725\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"
Sep 09 10:46:40.370244 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.370207    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dmmk\" (UniqueName: \"kubernetes.io/projected/48167dc3-0449-466b-8336-bb4c5a30f725-kube-api-access-6dmmk\") pod \"payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2\" (UID: \"48167dc3-0449-466b-8336-bb4c5a30f725\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"
Sep 09 10:46:40.503451 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.503415    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"
Sep 09 10:46:40.781489 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.781460    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"]
Sep 09 10:46:40.783138 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:46:40.783099    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48167dc3_0449_466b_8336_bb4c5a30f725.slice/crio-e065a82066db2149af7cf74da16ab734025577340e74baf9424f47763afb5136 WatchSource:0}: Error finding container e065a82066db2149af7cf74da16ab734025577340e74baf9424f47763afb5136: Status 404 returned error can't find the container with id e065a82066db2149af7cf74da16ab734025577340e74baf9424f47763afb5136
Sep 09 10:46:40.784249 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.784219    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:40.919437 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.919392    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8" event={"ID":"cc581664-0839-46c2-9607-05d064e65ca1","Type":"ContainerStarted","Data":"a36cf5222547f6ac7ebccccfb6717d1837cb525c8d3f928b10d72c7880fa53e6"}
Sep 09 10:46:40.919618 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.919467    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:40.919618 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.919494    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8" event={"ID":"cc581664-0839-46c2-9607-05d064e65ca1","Type":"ContainerStarted","Data":"e2935492d67af9d6a94b3555266ef2643a2c55d1d3beeb6a7e9ff9e61e51ca80"}
Sep 09 10:46:40.921121 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.921097    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2" event={"ID":"48167dc3-0449-466b-8336-bb4c5a30f725","Type":"ContainerStarted","Data":"65f061557bc259e7f6daf8a71dd21c3d7f382b47a2c37922913b05c1f171c3be"}
Sep 09 10:46:40.921256 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.921146    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"
Sep 09 10:46:40.921256 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.921173    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2" event={"ID":"48167dc3-0449-466b-8336-bb4c5a30f725","Type":"ContainerStarted","Data":"e065a82066db2149af7cf74da16ab734025577340e74baf9424f47763afb5136"}
Sep 09 10:46:40.922793 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.922754    2570 generic.go:356] "Generic (PLEG): container finished" podID="7de794cb-81be-4600-b724-e4ff6193ae3b" containerID="c9c63a447b8c052116e1b863a37489b5aa97ae7e20389fbda0d9b521ee4ab1cd" exitCode=0
Sep 09 10:46:40.922793 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.922786    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" event={"ID":"7de794cb-81be-4600-b724-e4ff6193ae3b","Type":"ContainerDied","Data":"c9c63a447b8c052116e1b863a37489b5aa97ae7e20389fbda0d9b521ee4ab1cd"}
Sep 09 10:46:40.922989 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.922815    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"
Sep 09 10:46:40.922989 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.922835    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47" event={"ID":"7de794cb-81be-4600-b724-e4ff6193ae3b","Type":"ContainerDied","Data":"7c94f0c056d97a64d45440f1f8f50b0cd65c9a68316dee5573099a66c377cd29"}
Sep 09 10:46:40.922989 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.922860    2570 scope.go:117] "RemoveContainer" containerID="c9c63a447b8c052116e1b863a37489b5aa97ae7e20389fbda0d9b521ee4ab1cd"
Sep 09 10:46:40.934198 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.934170    2570 scope.go:117] "RemoveContainer" containerID="fcf865eeee8e4bc0efcc60d422eeac49a4f6a7d189a2b1ebbc8314b08fa6d5aa"
Sep 09 10:46:40.941396 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.941319    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8" podStartSLOduration=1.413457111 podStartE2EDuration="1.941302203s" totalImagesPullingTime="527.845092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:46:40.937897894 +0000 UTC m=+1240.601536870" watchObservedRunningTime="2026-09-09 10:46:40.941302203 +0000 UTC m=+1240.604941179"
Sep 09 10:46:40.945478 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.945450    2570 scope.go:117] "RemoveContainer" containerID="c9c63a447b8c052116e1b863a37489b5aa97ae7e20389fbda0d9b521ee4ab1cd"
Sep 09 10:46:40.946604 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:46:40.946569    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c63a447b8c052116e1b863a37489b5aa97ae7e20389fbda0d9b521ee4ab1cd\": container with ID starting with c9c63a447b8c052116e1b863a37489b5aa97ae7e20389fbda0d9b521ee4ab1cd not found: ID does not exist" containerID="c9c63a447b8c052116e1b863a37489b5aa97ae7e20389fbda0d9b521ee4ab1cd"
Sep 09 10:46:40.946757 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.946661    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c63a447b8c052116e1b863a37489b5aa97ae7e20389fbda0d9b521ee4ab1cd"} err="failed to get container status \"c9c63a447b8c052116e1b863a37489b5aa97ae7e20389fbda0d9b521ee4ab1cd\": rpc error: code = NotFound desc = could not find container \"c9c63a447b8c052116e1b863a37489b5aa97ae7e20389fbda0d9b521ee4ab1cd\": container with ID starting with c9c63a447b8c052116e1b863a37489b5aa97ae7e20389fbda0d9b521ee4ab1cd not found: ID does not exist"
Sep 09 10:46:40.946757 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.946686    2570 scope.go:117] "RemoveContainer" containerID="fcf865eeee8e4bc0efcc60d422eeac49a4f6a7d189a2b1ebbc8314b08fa6d5aa"
Sep 09 10:46:40.951188 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:46:40.949098    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf865eeee8e4bc0efcc60d422eeac49a4f6a7d189a2b1ebbc8314b08fa6d5aa\": container with ID starting with fcf865eeee8e4bc0efcc60d422eeac49a4f6a7d189a2b1ebbc8314b08fa6d5aa not found: ID does not exist" containerID="fcf865eeee8e4bc0efcc60d422eeac49a4f6a7d189a2b1ebbc8314b08fa6d5aa"
Sep 09 10:46:40.951188 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.949136    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf865eeee8e4bc0efcc60d422eeac49a4f6a7d189a2b1ebbc8314b08fa6d5aa"} err="failed to get container status \"fcf865eeee8e4bc0efcc60d422eeac49a4f6a7d189a2b1ebbc8314b08fa6d5aa\": rpc error: code = NotFound desc = could not find container \"fcf865eeee8e4bc0efcc60d422eeac49a4f6a7d189a2b1ebbc8314b08fa6d5aa\": container with ID starting with fcf865eeee8e4bc0efcc60d422eeac49a4f6a7d189a2b1ebbc8314b08fa6d5aa not found: ID does not exist"
Sep 09 10:46:40.962758 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.962716    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2" podStartSLOduration=0.962700889 podStartE2EDuration="962.700889ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:46:40.960247094 +0000 UTC m=+1240.623886070" watchObservedRunningTime="2026-09-09 10:46:40.962700889 +0000 UTC m=+1240.626339862"
Sep 09 10:46:40.975169 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.975139    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"]
Sep 09 10:46:40.980919 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:40.980897    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dfb9af4b-openshift-default-6678fcd568-kck47"]
Sep 09 10:46:41.438464 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:41.438429    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:46:42.840093 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:42.840055    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7de794cb-81be-4600-b724-e4ff6193ae3b" path="/var/lib/kubelet/pods/7de794cb-81be-4600-b724-e4ff6193ae3b/volumes"
Sep 09 10:46:46.934886 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:46.934849    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:46:51.932492 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:51.932461    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"
Sep 09 10:46:57.017684 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:57.017646    2570 generic.go:356] "Generic (PLEG): container finished" podID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerID="0c72618f07734104e4579752da96cabdabf39a00df5fe443927e47f7d04f8215" exitCode=0
Sep 09 10:46:57.018246 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:57.017725    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6" event={"ID":"e07842b7-679c-4843-8b00-9aa214eefd5c","Type":"ContainerDied","Data":"0c72618f07734104e4579752da96cabdabf39a00df5fe443927e47f7d04f8215"}
Sep 09 10:46:57.018314 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:57.018294    2570 scope.go:117] "RemoveContainer" containerID="0c72618f07734104e4579752da96cabdabf39a00df5fe443927e47f7d04f8215"
Sep 09 10:46:57.019425 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:57.019397    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:46:57.019487 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:57.019472    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:46:57.019523 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:57.019505    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:46:58.025889 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:58.025850    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6" event={"ID":"e07842b7-679c-4843-8b00-9aa214eefd5c","Type":"ContainerStarted","Data":"6a29bbd45d145531bc1221fb48a60010c315bfd468d00589edc16e0318e55565"}
Sep 09 10:46:58.634449 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:58.634414    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:46:58.634633 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:58.634609    2570 patch_prober.go:28] interesting pod/e2e-derive-d84c5166-openshift-default-747db49656-dntb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 09 10:46:58.634685 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:58.634650    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6" podUID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 09 10:46:59.634295 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:59.634258    2570 patch_prober.go:28] interesting pod/e2e-derive-d84c5166-openshift-default-747db49656-dntb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 09 10:46:59.634697 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:46:59.634312    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6" podUID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 09 10:47:00.443221 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:00.443175    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:00.634576 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:00.634540    2570 patch_prober.go:28] interesting pod/e2e-derive-d84c5166-openshift-default-747db49656-dntb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 09 10:47:00.635009 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:00.634593    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6" podUID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 09 10:47:01.634667 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:01.634623    2570 patch_prober.go:28] interesting pod/e2e-derive-d84c5166-openshift-default-747db49656-dntb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 09 10:47:01.635192 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:01.634709    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6" podUID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 09 10:47:02.399881 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:02.399741    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:02.634369 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:02.634335    2570 patch_prober.go:28] interesting pod/e2e-derive-d84c5166-openshift-default-747db49656-dntb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 09 10:47:02.634536 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:02.634391    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6" podUID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 09 10:47:02.821560 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:02.821526    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:03.638456 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:03.638424    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:47:03.638822 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:03.638804    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:47:03.639591 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:03.639571    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:47:10.037454 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.037421    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"]
Sep 09 10:47:10.037869 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.037637    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8" podUID="cc581664-0839-46c2-9607-05d064e65ca1" containerName="maas-api" containerID="cri-o://a36cf5222547f6ac7ebccccfb6717d1837cb525c8d3f928b10d72c7880fa53e6" gracePeriod=30
Sep 09 10:47:10.082122 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.082092    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:10.137712 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.137683    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"]
Sep 09 10:47:10.137930 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.137905    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2" podUID="48167dc3-0449-466b-8336-bb4c5a30f725" containerName="payload-pre-processing" containerID="cri-o://65f061557bc259e7f6daf8a71dd21c3d7f382b47a2c37922913b05c1f171c3be" gracePeriod=30
Sep 09 10:47:10.214926 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.214889    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:10.307139 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.307105    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:47:10.376385 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.376347    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc581664-0839-46c2-9607-05d064e65ca1-maas-api-tls\") pod \"cc581664-0839-46c2-9607-05d064e65ca1\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") "
Sep 09 10:47:10.376385 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.376392    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc581664-0839-46c2-9607-05d064e65ca1-metrics-certs\") pod \"cc581664-0839-46c2-9607-05d064e65ca1\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") "
Sep 09 10:47:10.376629 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.376463    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc581664-0839-46c2-9607-05d064e65ca1-openshift-service-ca\") pod \"cc581664-0839-46c2-9607-05d064e65ca1\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") "
Sep 09 10:47:10.376629 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.376509    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5g4q\" (UniqueName: \"kubernetes.io/projected/cc581664-0839-46c2-9607-05d064e65ca1-kube-api-access-s5g4q\") pod \"cc581664-0839-46c2-9607-05d064e65ca1\" (UID: \"cc581664-0839-46c2-9607-05d064e65ca1\") "
Sep 09 10:47:10.376880 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.376857    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc581664-0839-46c2-9607-05d064e65ca1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc581664-0839-46c2-9607-05d064e65ca1" (UID: "cc581664-0839-46c2-9607-05d064e65ca1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:47:10.379422 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.379375    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc581664-0839-46c2-9607-05d064e65ca1-kube-api-access-s5g4q" (OuterVolumeSpecName: "kube-api-access-s5g4q") pod "cc581664-0839-46c2-9607-05d064e65ca1" (UID: "cc581664-0839-46c2-9607-05d064e65ca1"). InnerVolumeSpecName "kube-api-access-s5g4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:47:10.379530 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.379464    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc581664-0839-46c2-9607-05d064e65ca1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc581664-0839-46c2-9607-05d064e65ca1" (UID: "cc581664-0839-46c2-9607-05d064e65ca1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:47:10.379530 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.379474    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc581664-0839-46c2-9607-05d064e65ca1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc581664-0839-46c2-9607-05d064e65ca1" (UID: "cc581664-0839-46c2-9607-05d064e65ca1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:47:10.417879 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.417382    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:10.429650 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.429624    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"
Sep 09 10:47:10.489110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.489058    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48167dc3-0449-466b-8336-bb4c5a30f725-plugins-config-volume\") pod \"48167dc3-0449-466b-8336-bb4c5a30f725\" (UID: \"48167dc3-0449-466b-8336-bb4c5a30f725\") "
Sep 09 10:47:10.489308 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.489138    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dmmk\" (UniqueName: \"kubernetes.io/projected/48167dc3-0449-466b-8336-bb4c5a30f725-kube-api-access-6dmmk\") pod \"48167dc3-0449-466b-8336-bb4c5a30f725\" (UID: \"48167dc3-0449-466b-8336-bb4c5a30f725\") "
Sep 09 10:47:10.491382 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.491348    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc581664-0839-46c2-9607-05d064e65ca1-maas-api-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:10.491382 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.491382    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc581664-0839-46c2-9607-05d064e65ca1-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:10.491581 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.491403    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc581664-0839-46c2-9607-05d064e65ca1-openshift-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:10.491581 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.491418    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5g4q\" (UniqueName: \"kubernetes.io/projected/cc581664-0839-46c2-9607-05d064e65ca1-kube-api-access-s5g4q\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:10.491666 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.491601    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48167dc3-0449-466b-8336-bb4c5a30f725-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "48167dc3-0449-466b-8336-bb4c5a30f725" (UID: "48167dc3-0449-466b-8336-bb4c5a30f725"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:47:10.497487 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.497447    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48167dc3-0449-466b-8336-bb4c5a30f725-kube-api-access-6dmmk" (OuterVolumeSpecName: "kube-api-access-6dmmk") pod "48167dc3-0449-466b-8336-bb4c5a30f725" (UID: "48167dc3-0449-466b-8336-bb4c5a30f725"). InnerVolumeSpecName "kube-api-access-6dmmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:47:10.512784 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.512743    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:10.592425 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.592331    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48167dc3-0449-466b-8336-bb4c5a30f725-plugins-config-volume\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:10.592425 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.592365    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dmmk\" (UniqueName: \"kubernetes.io/projected/48167dc3-0449-466b-8336-bb4c5a30f725-kube-api-access-6dmmk\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:10.613565 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.613534    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:10.670431 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.670402    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"]
Sep 09 10:47:10.670707 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.670662    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6" podUID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerName="istio-proxy" containerID="cri-o://6a29bbd45d145531bc1221fb48a60010c315bfd468d00589edc16e0318e55565" gracePeriod=30
Sep 09 10:47:10.697924 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:10.697887    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:11.102077 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.101976    2570 generic.go:356] "Generic (PLEG): container finished" podID="cc581664-0839-46c2-9607-05d064e65ca1" containerID="a36cf5222547f6ac7ebccccfb6717d1837cb525c8d3f928b10d72c7880fa53e6" exitCode=0
Sep 09 10:47:11.102077 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.102040    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"
Sep 09 10:47:11.102077 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.102053    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8" event={"ID":"cc581664-0839-46c2-9607-05d064e65ca1","Type":"ContainerDied","Data":"a36cf5222547f6ac7ebccccfb6717d1837cb525c8d3f928b10d72c7880fa53e6"}
Sep 09 10:47:11.102661 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.102106    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8" event={"ID":"cc581664-0839-46c2-9607-05d064e65ca1","Type":"ContainerDied","Data":"e2935492d67af9d6a94b3555266ef2643a2c55d1d3beeb6a7e9ff9e61e51ca80"}
Sep 09 10:47:11.102661 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.102134    2570 scope.go:117] "RemoveContainer" containerID="a36cf5222547f6ac7ebccccfb6717d1837cb525c8d3f928b10d72c7880fa53e6"
Sep 09 10:47:11.103637 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.103582    2570 generic.go:356] "Generic (PLEG): container finished" podID="48167dc3-0449-466b-8336-bb4c5a30f725" containerID="65f061557bc259e7f6daf8a71dd21c3d7f382b47a2c37922913b05c1f171c3be" exitCode=0
Sep 09 10:47:11.103637 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.103617    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2" event={"ID":"48167dc3-0449-466b-8336-bb4c5a30f725","Type":"ContainerDied","Data":"65f061557bc259e7f6daf8a71dd21c3d7f382b47a2c37922913b05c1f171c3be"}
Sep 09 10:47:11.103809 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.103645    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"
Sep 09 10:47:11.103809 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.103655    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2" event={"ID":"48167dc3-0449-466b-8336-bb4c5a30f725","Type":"ContainerDied","Data":"e065a82066db2149af7cf74da16ab734025577340e74baf9424f47763afb5136"}
Sep 09 10:47:11.111543 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.111525    2570 scope.go:117] "RemoveContainer" containerID="a36cf5222547f6ac7ebccccfb6717d1837cb525c8d3f928b10d72c7880fa53e6"
Sep 09 10:47:11.111862 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:47:11.111838    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a36cf5222547f6ac7ebccccfb6717d1837cb525c8d3f928b10d72c7880fa53e6\": container with ID starting with a36cf5222547f6ac7ebccccfb6717d1837cb525c8d3f928b10d72c7880fa53e6 not found: ID does not exist" containerID="a36cf5222547f6ac7ebccccfb6717d1837cb525c8d3f928b10d72c7880fa53e6"
Sep 09 10:47:11.111929 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.111878    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a36cf5222547f6ac7ebccccfb6717d1837cb525c8d3f928b10d72c7880fa53e6"} err="failed to get container status \"a36cf5222547f6ac7ebccccfb6717d1837cb525c8d3f928b10d72c7880fa53e6\": rpc error: code = NotFound desc = could not find container \"a36cf5222547f6ac7ebccccfb6717d1837cb525c8d3f928b10d72c7880fa53e6\": container with ID starting with a36cf5222547f6ac7ebccccfb6717d1837cb525c8d3f928b10d72c7880fa53e6 not found: ID does not exist"
Sep 09 10:47:11.111929 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.111902    2570 scope.go:117] "RemoveContainer" containerID="65f061557bc259e7f6daf8a71dd21c3d7f382b47a2c37922913b05c1f171c3be"
Sep 09 10:47:11.120010 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.119992    2570 scope.go:117] "RemoveContainer" containerID="65f061557bc259e7f6daf8a71dd21c3d7f382b47a2c37922913b05c1f171c3be"
Sep 09 10:47:11.120292 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:47:11.120270    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f061557bc259e7f6daf8a71dd21c3d7f382b47a2c37922913b05c1f171c3be\": container with ID starting with 65f061557bc259e7f6daf8a71dd21c3d7f382b47a2c37922913b05c1f171c3be not found: ID does not exist" containerID="65f061557bc259e7f6daf8a71dd21c3d7f382b47a2c37922913b05c1f171c3be"
Sep 09 10:47:11.120362 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.120298    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f061557bc259e7f6daf8a71dd21c3d7f382b47a2c37922913b05c1f171c3be"} err="failed to get container status \"65f061557bc259e7f6daf8a71dd21c3d7f382b47a2c37922913b05c1f171c3be\": rpc error: code = NotFound desc = could not find container \"65f061557bc259e7f6daf8a71dd21c3d7f382b47a2c37922913b05c1f171c3be\": container with ID starting with 65f061557bc259e7f6daf8a71dd21c3d7f382b47a2c37922913b05c1f171c3be not found: ID does not exist"
Sep 09 10:47:11.125430 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.125405    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"]
Sep 09 10:47:11.133005 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.132984    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d84c5166-7574984658-6lzg2"]
Sep 09 10:47:11.144949 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.144923    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"]
Sep 09 10:47:11.151291 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.151267    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d84c5166-7c67fd8d55-msvf8"]
Sep 09 10:47:11.396472 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.396390    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:11.562226 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.562192    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:11.683889 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:11.683783    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:12.840290 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:12.840259    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48167dc3-0449-466b-8336-bb4c5a30f725" path="/var/lib/kubelet/pods/48167dc3-0449-466b-8336-bb4c5a30f725/volumes"
Sep 09 10:47:12.840749 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:12.840679    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc581664-0839-46c2-9607-05d064e65ca1" path="/var/lib/kubelet/pods/cc581664-0839-46c2-9607-05d064e65ca1/volumes"
Sep 09 10:47:12.988232 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:12.988200    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:15.930917 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:15.930890    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:47:16.044800 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.044738    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-workload-certs\") pod \"e07842b7-679c-4843-8b00-9aa214eefd5c\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") "
Sep 09 10:47:16.044800 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.044808    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-credential-socket\") pod \"e07842b7-679c-4843-8b00-9aa214eefd5c\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") "
Sep 09 10:47:16.045030 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.044837    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-data\") pod \"e07842b7-679c-4843-8b00-9aa214eefd5c\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") "
Sep 09 10:47:16.045030 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.044870    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-token\") pod \"e07842b7-679c-4843-8b00-9aa214eefd5c\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") "
Sep 09 10:47:16.045030 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.044911    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-podinfo\") pod \"e07842b7-679c-4843-8b00-9aa214eefd5c\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") "
Sep 09 10:47:16.045030 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.044937    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsrvx\" (UniqueName: \"kubernetes.io/projected/e07842b7-679c-4843-8b00-9aa214eefd5c-kube-api-access-hsrvx\") pod \"e07842b7-679c-4843-8b00-9aa214eefd5c\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") "
Sep 09 10:47:16.045030 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.044974    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-envoy\") pod \"e07842b7-679c-4843-8b00-9aa214eefd5c\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") "
Sep 09 10:47:16.045309 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.045044    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-workload-socket\") pod \"e07842b7-679c-4843-8b00-9aa214eefd5c\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") "
Sep 09 10:47:16.045309 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.045052    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e07842b7-679c-4843-8b00-9aa214eefd5c" (UID: "e07842b7-679c-4843-8b00-9aa214eefd5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:47:16.045309 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.045094    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07842b7-679c-4843-8b00-9aa214eefd5c-istiod-ca-cert\") pod \"e07842b7-679c-4843-8b00-9aa214eefd5c\" (UID: \"e07842b7-679c-4843-8b00-9aa214eefd5c\") "
Sep 09 10:47:16.045309 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.045192    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-data" (OuterVolumeSpecName: "istio-data") pod "e07842b7-679c-4843-8b00-9aa214eefd5c" (UID: "e07842b7-679c-4843-8b00-9aa214eefd5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:47:16.045524 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.045153    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e07842b7-679c-4843-8b00-9aa214eefd5c" (UID: "e07842b7-679c-4843-8b00-9aa214eefd5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:47:16.045524 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.045377    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e07842b7-679c-4843-8b00-9aa214eefd5c" (UID: "e07842b7-679c-4843-8b00-9aa214eefd5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:47:16.045630 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.045526    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-credential-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:16.045630 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.045548    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-data\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:16.045630 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.045561    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-workload-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:16.045630 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.045575    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-workload-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:16.045630 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.045572    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07842b7-679c-4843-8b00-9aa214eefd5c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e07842b7-679c-4843-8b00-9aa214eefd5c" (UID: "e07842b7-679c-4843-8b00-9aa214eefd5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:47:16.047261 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.047229    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e07842b7-679c-4843-8b00-9aa214eefd5c" (UID: "e07842b7-679c-4843-8b00-9aa214eefd5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:47:16.047375 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.047267    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e07842b7-679c-4843-8b00-9aa214eefd5c" (UID: "e07842b7-679c-4843-8b00-9aa214eefd5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:47:16.047604 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.047584    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-token" (OuterVolumeSpecName: "istio-token") pod "e07842b7-679c-4843-8b00-9aa214eefd5c" (UID: "e07842b7-679c-4843-8b00-9aa214eefd5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:47:16.047689 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.047614    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07842b7-679c-4843-8b00-9aa214eefd5c-kube-api-access-hsrvx" (OuterVolumeSpecName: "kube-api-access-hsrvx") pod "e07842b7-679c-4843-8b00-9aa214eefd5c" (UID: "e07842b7-679c-4843-8b00-9aa214eefd5c"). InnerVolumeSpecName "kube-api-access-hsrvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:47:16.134026 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.133991    2570 generic.go:356] "Generic (PLEG): container finished" podID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerID="6a29bbd45d145531bc1221fb48a60010c315bfd468d00589edc16e0318e55565" exitCode=0
Sep 09 10:47:16.134212 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.134050    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"
Sep 09 10:47:16.134212 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.134078    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6" event={"ID":"e07842b7-679c-4843-8b00-9aa214eefd5c","Type":"ContainerDied","Data":"6a29bbd45d145531bc1221fb48a60010c315bfd468d00589edc16e0318e55565"}
Sep 09 10:47:16.134212 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.134148    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6" event={"ID":"e07842b7-679c-4843-8b00-9aa214eefd5c","Type":"ContainerDied","Data":"34de440a1b46a3b115768108cee59e0c6e32b4c53df9fbc29f176d5f1c36c0a4"}
Sep 09 10:47:16.134212 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.134178    2570 scope.go:117] "RemoveContainer" containerID="6a29bbd45d145531bc1221fb48a60010c315bfd468d00589edc16e0318e55565"
Sep 09 10:47:16.143402 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.143385    2570 scope.go:117] "RemoveContainer" containerID="0c72618f07734104e4579752da96cabdabf39a00df5fe443927e47f7d04f8215"
Sep 09 10:47:16.146429 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.146408    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:16.146511 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.146436    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-podinfo\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:16.146511 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.146451    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsrvx\" (UniqueName: \"kubernetes.io/projected/e07842b7-679c-4843-8b00-9aa214eefd5c-kube-api-access-hsrvx\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:16.146511 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.146469    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07842b7-679c-4843-8b00-9aa214eefd5c-istio-envoy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:16.146511 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.146482    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07842b7-679c-4843-8b00-9aa214eefd5c-istiod-ca-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:47:16.151728 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.151709    2570 scope.go:117] "RemoveContainer" containerID="6a29bbd45d145531bc1221fb48a60010c315bfd468d00589edc16e0318e55565"
Sep 09 10:47:16.152036 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:47:16.152019    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a29bbd45d145531bc1221fb48a60010c315bfd468d00589edc16e0318e55565\": container with ID starting with 6a29bbd45d145531bc1221fb48a60010c315bfd468d00589edc16e0318e55565 not found: ID does not exist" containerID="6a29bbd45d145531bc1221fb48a60010c315bfd468d00589edc16e0318e55565"
Sep 09 10:47:16.152084 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.152046    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a29bbd45d145531bc1221fb48a60010c315bfd468d00589edc16e0318e55565"} err="failed to get container status \"6a29bbd45d145531bc1221fb48a60010c315bfd468d00589edc16e0318e55565\": rpc error: code = NotFound desc = could not find container \"6a29bbd45d145531bc1221fb48a60010c315bfd468d00589edc16e0318e55565\": container with ID starting with 6a29bbd45d145531bc1221fb48a60010c315bfd468d00589edc16e0318e55565 not found: ID does not exist"
Sep 09 10:47:16.152084 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.152065    2570 scope.go:117] "RemoveContainer" containerID="0c72618f07734104e4579752da96cabdabf39a00df5fe443927e47f7d04f8215"
Sep 09 10:47:16.152309 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:47:16.152292    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c72618f07734104e4579752da96cabdabf39a00df5fe443927e47f7d04f8215\": container with ID starting with 0c72618f07734104e4579752da96cabdabf39a00df5fe443927e47f7d04f8215 not found: ID does not exist" containerID="0c72618f07734104e4579752da96cabdabf39a00df5fe443927e47f7d04f8215"
Sep 09 10:47:16.152355 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.152314    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c72618f07734104e4579752da96cabdabf39a00df5fe443927e47f7d04f8215"} err="failed to get container status \"0c72618f07734104e4579752da96cabdabf39a00df5fe443927e47f7d04f8215\": rpc error: code = NotFound desc = could not find container \"0c72618f07734104e4579752da96cabdabf39a00df5fe443927e47f7d04f8215\": container with ID starting with 0c72618f07734104e4579752da96cabdabf39a00df5fe443927e47f7d04f8215 not found: ID does not exist"
Sep 09 10:47:16.159282 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.159259    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"]
Sep 09 10:47:16.164481 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.164460    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d84c5166-openshift-default-747db49656-dntb6"]
Sep 09 10:47:16.839699 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:16.839667    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e07842b7-679c-4843-8b00-9aa214eefd5c" path="/var/lib/kubelet/pods/e07842b7-679c-4843-8b00-9aa214eefd5c/volumes"
Sep 09 10:47:18.263370 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:18.263335    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:18.522514 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:18.522422    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:18.970237 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:18.970151    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:19.238648 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.238613    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"]
Sep 09 10:47:19.239251 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.239235    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerName="istio-proxy"
Sep 09 10:47:19.239300 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.239256    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerName="istio-proxy"
Sep 09 10:47:19.239336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.239300    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc581664-0839-46c2-9607-05d064e65ca1" containerName="maas-api"
Sep 09 10:47:19.239336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.239309    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc581664-0839-46c2-9607-05d064e65ca1" containerName="maas-api"
Sep 09 10:47:19.239336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.239325    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48167dc3-0449-466b-8336-bb4c5a30f725" containerName="payload-pre-processing"
Sep 09 10:47:19.239336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.239334    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="48167dc3-0449-466b-8336-bb4c5a30f725" containerName="payload-pre-processing"
Sep 09 10:47:19.239463 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.239455    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerName="istio-proxy"
Sep 09 10:47:19.239512 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.239502    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc581664-0839-46c2-9607-05d064e65ca1" containerName="maas-api"
Sep 09 10:47:19.239562 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.239513    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="48167dc3-0449-466b-8336-bb4c5a30f725" containerName="payload-pre-processing"
Sep 09 10:47:19.244333 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.244310    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.247583 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.247552    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-928d2a-openshift-default-dockercfg-4pjm2\""
Sep 09 10:47:19.259758 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.259731    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"]
Sep 09 10:47:19.269650 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.269624    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:19.380872 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.380837    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f94307-2cbc-4a82-b1ca-240223286467-istio-podinfo\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.381097 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.380881    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f94307-2cbc-4a82-b1ca-240223286467-istio-token\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.381097 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.380906    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-istio-envoy\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.381097 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.380981    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-workload-socket\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.381097 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.381030    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f94307-2cbc-4a82-b1ca-240223286467-istiod-ca-cert\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.381247 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.381116    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-workload-certs\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.381247 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.381142    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwfc8\" (UniqueName: \"kubernetes.io/projected/f7f94307-2cbc-4a82-b1ca-240223286467-kube-api-access-rwfc8\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.381247 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.381160    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-istio-data\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.381247 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.381190    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-credential-socket\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.481936 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.481894    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-credential-socket\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.481936 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.481958    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f94307-2cbc-4a82-b1ca-240223286467-istio-podinfo\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.482379 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.482005    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f94307-2cbc-4a82-b1ca-240223286467-istio-token\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.482379 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.482039    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-istio-envoy\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.482379 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.482070    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-workload-socket\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.482379 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.482115    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f94307-2cbc-4a82-b1ca-240223286467-istiod-ca-cert\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.482379 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.482217    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-workload-certs\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.482379 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.482256    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwfc8\" (UniqueName: \"kubernetes.io/projected/f7f94307-2cbc-4a82-b1ca-240223286467-kube-api-access-rwfc8\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.482379 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.482283    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-istio-data\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.482379 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.482376    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-credential-socket\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.482840 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.482663    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-workload-certs\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.482840 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.482752    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-workload-socket\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.482962 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.482884    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-istio-data\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.483084 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.483061    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f94307-2cbc-4a82-b1ca-240223286467-istiod-ca-cert\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.484583 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.484552    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-istio-envoy\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.484755 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.484739    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f94307-2cbc-4a82-b1ca-240223286467-istio-podinfo\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.496996 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.496934    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwfc8\" (UniqueName: \"kubernetes.io/projected/f7f94307-2cbc-4a82-b1ca-240223286467-kube-api-access-rwfc8\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.506187 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.506166    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f94307-2cbc-4a82-b1ca-240223286467-istio-token\") pod \"e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") " pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.555385 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.555348    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:19.566736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.566712    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:19.837441 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.837411    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"]
Sep 09 10:47:19.840177 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:47:19.840147    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f94307_2cbc_4a82_b1ca_240223286467.slice/crio-656d2d0db3be61d2a0c84ce75aae34efbd10327addaac53583e6b40942f69246 WatchSource:0}: Error finding container 656d2d0db3be61d2a0c84ce75aae34efbd10327addaac53583e6b40942f69246: Status 404 returned error can't find the container with id 656d2d0db3be61d2a0c84ce75aae34efbd10327addaac53583e6b40942f69246
Sep 09 10:47:19.842347 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.842315    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:47:19.842430 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.842382    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:47:19.842430 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:19.842411    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:47:20.095630 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:20.095533    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:20.159621 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:20.159586    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4" event={"ID":"f7f94307-2cbc-4a82-b1ca-240223286467","Type":"ContainerStarted","Data":"b931af0997f156c8fb04ff21ae384ca85a2dd56880745ece96c46e039c3ae161"}
Sep 09 10:47:20.159865 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:20.159693    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4" event={"ID":"f7f94307-2cbc-4a82-b1ca-240223286467","Type":"ContainerStarted","Data":"656d2d0db3be61d2a0c84ce75aae34efbd10327addaac53583e6b40942f69246"}
Sep 09 10:47:20.195405 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:20.195344    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4" podStartSLOduration=1.1953245909999999 podStartE2EDuration="1.195324591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:47:20.191896126 +0000 UTC m=+1279.855535103" watchObservedRunningTime="2026-09-09 10:47:20.195324591 +0000 UTC m=+1279.858963565"
Sep 09 10:47:20.209563 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:20.209504    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:20.308968 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:20.308928    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:20.407828 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:20.407698    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:20.555709 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:20.555673    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:20.561032 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:20.561006    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:20.790612 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:20.790576    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:21.165671 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:21.165575    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:21.166642 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:21.166623    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:22.124295 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:22.124241    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:22.220684 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:22.220649    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:23.413811 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.413775    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"]
Sep 09 10:47:23.414214 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.414195    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerName="istio-proxy"
Sep 09 10:47:23.414214 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.414207    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerName="istio-proxy"
Sep 09 10:47:23.414308 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.414296    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e07842b7-679c-4843-8b00-9aa214eefd5c" containerName="istio-proxy"
Sep 09 10:47:23.419926 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.418838    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:23.425214 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.423065    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-928d2a-serving-cert\""
Sep 09 10:47:23.432804 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.432359    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"]
Sep 09 10:47:23.467042 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.467007    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:23.527613 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.527574    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc47t\" (UniqueName: \"kubernetes.io/projected/e194eeed-28db-40df-aead-dfcded573d32-kube-api-access-rc47t\") pod \"maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:23.527613 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.527617    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e194eeed-28db-40df-aead-dfcded573d32-maas-api-tls\") pod \"maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:23.527880 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.527710    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e194eeed-28db-40df-aead-dfcded573d32-openshift-service-ca\") pod \"maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:23.527880 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.527773    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e194eeed-28db-40df-aead-dfcded573d32-metrics-certs\") pod \"maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:23.561930 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.561896    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:23.629352 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.629311    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc47t\" (UniqueName: \"kubernetes.io/projected/e194eeed-28db-40df-aead-dfcded573d32-kube-api-access-rc47t\") pod \"maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:23.629505 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.629358    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e194eeed-28db-40df-aead-dfcded573d32-maas-api-tls\") pod \"maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:23.629505 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.629401    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e194eeed-28db-40df-aead-dfcded573d32-openshift-service-ca\") pod \"maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:23.629505 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.629432    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e194eeed-28db-40df-aead-dfcded573d32-metrics-certs\") pod \"maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:23.630202 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.630179    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e194eeed-28db-40df-aead-dfcded573d32-openshift-service-ca\") pod \"maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:23.632013 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.631978    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e194eeed-28db-40df-aead-dfcded573d32-metrics-certs\") pod \"maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:23.632105 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.632020    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e194eeed-28db-40df-aead-dfcded573d32-maas-api-tls\") pod \"maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:23.642618 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.642591    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc47t\" (UniqueName: \"kubernetes.io/projected/e194eeed-28db-40df-aead-dfcded573d32-kube-api-access-rc47t\") pod \"maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:23.660264 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.660235    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:23.737543 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:23.737452    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:24.030156 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:24.030127    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"]
Sep 09 10:47:24.031888 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:47:24.031861    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode194eeed_28db_40df_aead_dfcded573d32.slice/crio-e28ee5f6f2b13d0a2080d3ea9ef1726c6ea4c17933d0288897c6963478b501c8 WatchSource:0}: Error finding container e28ee5f6f2b13d0a2080d3ea9ef1726c6ea4c17933d0288897c6963478b501c8: Status 404 returned error can't find the container with id e28ee5f6f2b13d0a2080d3ea9ef1726c6ea4c17933d0288897c6963478b501c8
Sep 09 10:47:24.183307 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:24.183265    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb" event={"ID":"e194eeed-28db-40df-aead-dfcded573d32","Type":"ContainerStarted","Data":"e28ee5f6f2b13d0a2080d3ea9ef1726c6ea4c17933d0288897c6963478b501c8"}
Sep 09 10:47:24.212782 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:24.212725    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:24.313027 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:24.312944    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:25.190526 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:25.190488    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb" event={"ID":"e194eeed-28db-40df-aead-dfcded573d32","Type":"ContainerStarted","Data":"e7235a6a1e778abf7be25604c8c679db5fa7365291548ebbecddd0f8c3153da8"}
Sep 09 10:47:25.190965 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:25.190619    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:25.210736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:25.210678    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb" podStartSLOduration=1.528949109 podStartE2EDuration="2.21066169s" totalImagesPullingTime="681.712581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:47:25.209075878 +0000 UTC m=+1284.872714853" watchObservedRunningTime="2026-09-09 10:47:25.21066169 +0000 UTC m=+1284.874300664"
Sep 09 10:47:31.202565 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:31.202531    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:47:41.282425 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:41.282390    2570 generic.go:356] "Generic (PLEG): container finished" podID="f7f94307-2cbc-4a82-b1ca-240223286467" containerID="b931af0997f156c8fb04ff21ae384ca85a2dd56880745ece96c46e039c3ae161" exitCode=0
Sep 09 10:47:41.282854 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:41.282462    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4" event={"ID":"f7f94307-2cbc-4a82-b1ca-240223286467","Type":"ContainerDied","Data":"b931af0997f156c8fb04ff21ae384ca85a2dd56880745ece96c46e039c3ae161"}
Sep 09 10:47:41.282987 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:41.282970    2570 scope.go:117] "RemoveContainer" containerID="b931af0997f156c8fb04ff21ae384ca85a2dd56880745ece96c46e039c3ae161"
Sep 09 10:47:41.284177 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:41.284144    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:47:41.284270 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:41.284212    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:47:41.284270 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:41.284245    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:47:42.289540 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:42.289505    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4" event={"ID":"f7f94307-2cbc-4a82-b1ca-240223286467","Type":"ContainerStarted","Data":"9642a0af005cb45fdd125c96dba426c0a472ddeabee1f10bb7d618cc7f35c23f"}
Sep 09 10:47:42.556635 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:42.556527    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:42.557065 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:42.557018    2570 patch_prober.go:28] interesting pod/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 10:47:42.557335 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:42.557297    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4" podUID="f7f94307-2cbc-4a82-b1ca-240223286467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 10:47:43.556727 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:43.556683    2570 patch_prober.go:28] interesting pod/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 10:47:43.557396 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:43.557362    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4" podUID="f7f94307-2cbc-4a82-b1ca-240223286467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 10:47:44.556439 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:44.556402    2570 patch_prober.go:28] interesting pod/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 10:47:44.556879 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:44.556842    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4" podUID="f7f94307-2cbc-4a82-b1ca-240223286467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 10:47:45.556108 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:45.556067    2570 patch_prober.go:28] interesting pod/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 10:47:45.556434 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:45.556393    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4" podUID="f7f94307-2cbc-4a82-b1ca-240223286467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 10:47:45.917196 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:45.917083    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:46.556403 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:46.556348    2570 patch_prober.go:28] interesting pod/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 10:47:46.556633 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:46.556427    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4" podUID="f7f94307-2cbc-4a82-b1ca-240223286467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 10:47:46.717587 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:46.717548    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:47.305309 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:47.305277    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:47.615063 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:47.614958    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:47.615510 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:47.615463    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:47.616345 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:47.616320    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:47:53.655444 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:53.655406    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:53.767285 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:53.767246    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:47:53.867591 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:47:53.867557    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:01.345489 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:01.345451    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:03.673248 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:03.673209    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:09.686518 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:09.686478    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:10.303572 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:10.303533    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"]
Sep 09 10:48:10.328757 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:10.328725    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:11.378431 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:11.378386    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:11.628879 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:11.628792    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:13.925493 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:13.925459    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:14.511447 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.511412    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"]
Sep 09 10:48:14.511689 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.511656    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb" podUID="e194eeed-28db-40df-aead-dfcded573d32" containerName="maas-api" containerID="cri-o://e7235a6a1e778abf7be25604c8c679db5fa7365291548ebbecddd0f8c3153da8" gracePeriod=30
Sep 09 10:48:14.539007 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.538971    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:14.607751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.607723    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:14.706751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.706706    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:14.767121 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.767057    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:48:14.779306 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.779281    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:14.870680 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.870646    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc47t\" (UniqueName: \"kubernetes.io/projected/e194eeed-28db-40df-aead-dfcded573d32-kube-api-access-rc47t\") pod \"e194eeed-28db-40df-aead-dfcded573d32\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") "
Sep 09 10:48:14.870888 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.870703    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e194eeed-28db-40df-aead-dfcded573d32-openshift-service-ca\") pod \"e194eeed-28db-40df-aead-dfcded573d32\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") "
Sep 09 10:48:14.870888 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.870842    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e194eeed-28db-40df-aead-dfcded573d32-metrics-certs\") pod \"e194eeed-28db-40df-aead-dfcded573d32\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") "
Sep 09 10:48:14.870888 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.870865    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e194eeed-28db-40df-aead-dfcded573d32-maas-api-tls\") pod \"e194eeed-28db-40df-aead-dfcded573d32\" (UID: \"e194eeed-28db-40df-aead-dfcded573d32\") "
Sep 09 10:48:14.871182 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.871150    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e194eeed-28db-40df-aead-dfcded573d32-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e194eeed-28db-40df-aead-dfcded573d32" (UID: "e194eeed-28db-40df-aead-dfcded573d32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:48:14.873009 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.872974    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e194eeed-28db-40df-aead-dfcded573d32-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e194eeed-28db-40df-aead-dfcded573d32" (UID: "e194eeed-28db-40df-aead-dfcded573d32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:48:14.873336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.873304    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e194eeed-28db-40df-aead-dfcded573d32-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e194eeed-28db-40df-aead-dfcded573d32" (UID: "e194eeed-28db-40df-aead-dfcded573d32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:48:14.873424 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.873397    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e194eeed-28db-40df-aead-dfcded573d32-kube-api-access-rc47t" (OuterVolumeSpecName: "kube-api-access-rc47t") pod "e194eeed-28db-40df-aead-dfcded573d32" (UID: "e194eeed-28db-40df-aead-dfcded573d32"). InnerVolumeSpecName "kube-api-access-rc47t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:48:14.972126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.972080    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e194eeed-28db-40df-aead-dfcded573d32-openshift-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:14.972126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.972115    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e194eeed-28db-40df-aead-dfcded573d32-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:14.972126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.972124    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e194eeed-28db-40df-aead-dfcded573d32-maas-api-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:14.972126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:14.972132    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc47t\" (UniqueName: \"kubernetes.io/projected/e194eeed-28db-40df-aead-dfcded573d32-kube-api-access-rc47t\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:15.105538 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:15.105457    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:15.460929 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:15.460834    2570 generic.go:356] "Generic (PLEG): container finished" podID="e194eeed-28db-40df-aead-dfcded573d32" containerID="e7235a6a1e778abf7be25604c8c679db5fa7365291548ebbecddd0f8c3153da8" exitCode=0
Sep 09 10:48:15.460929 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:15.460877    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"
Sep 09 10:48:15.460929 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:15.460910    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb" event={"ID":"e194eeed-28db-40df-aead-dfcded573d32","Type":"ContainerDied","Data":"e7235a6a1e778abf7be25604c8c679db5fa7365291548ebbecddd0f8c3153da8"}
Sep 09 10:48:15.461196 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:15.460976    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb" event={"ID":"e194eeed-28db-40df-aead-dfcded573d32","Type":"ContainerDied","Data":"e28ee5f6f2b13d0a2080d3ea9ef1726c6ea4c17933d0288897c6963478b501c8"}
Sep 09 10:48:15.461196 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:15.461006    2570 scope.go:117] "RemoveContainer" containerID="e7235a6a1e778abf7be25604c8c679db5fa7365291548ebbecddd0f8c3153da8"
Sep 09 10:48:15.470564 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:15.470540    2570 scope.go:117] "RemoveContainer" containerID="e7235a6a1e778abf7be25604c8c679db5fa7365291548ebbecddd0f8c3153da8"
Sep 09 10:48:15.470849 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:48:15.470826    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7235a6a1e778abf7be25604c8c679db5fa7365291548ebbecddd0f8c3153da8\": container with ID starting with e7235a6a1e778abf7be25604c8c679db5fa7365291548ebbecddd0f8c3153da8 not found: ID does not exist" containerID="e7235a6a1e778abf7be25604c8c679db5fa7365291548ebbecddd0f8c3153da8"
Sep 09 10:48:15.470950 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:15.470856    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7235a6a1e778abf7be25604c8c679db5fa7365291548ebbecddd0f8c3153da8"} err="failed to get container status \"e7235a6a1e778abf7be25604c8c679db5fa7365291548ebbecddd0f8c3153da8\": rpc error: code = NotFound desc = could not find container \"e7235a6a1e778abf7be25604c8c679db5fa7365291548ebbecddd0f8c3153da8\": container with ID starting with e7235a6a1e778abf7be25604c8c679db5fa7365291548ebbecddd0f8c3153da8 not found: ID does not exist"
Sep 09 10:48:15.482805 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:15.482780    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"]
Sep 09 10:48:15.486994 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:15.486967    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-928d2a-56f64dc5f5-678cb"]
Sep 09 10:48:16.839664 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:16.839628    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e194eeed-28db-40df-aead-dfcded573d32" path="/var/lib/kubelet/pods/e194eeed-28db-40df-aead-dfcded573d32/volumes"
Sep 09 10:48:17.034344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:17.034307    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:17.214853 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:17.214728    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:17.370265 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:17.370231    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:20.183290 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:20.183241    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:21.718053 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:21.718018    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:22.333108 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.333060    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"]
Sep 09 10:48:22.333416 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.333386    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc" podUID="3b1056f9-b34f-4cb7-994f-5338cfbf7579" containerName="maas-api" containerID="cri-o://9937beacfb2360f1c0f1b804cf3c1f24090939e63648ba77f0f67717d87020ca" gracePeriod=30
Sep 09 10:48:22.368605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.368575    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:22.467414 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.467375    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:22.510366 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.510332    2570 generic.go:356] "Generic (PLEG): container finished" podID="3b1056f9-b34f-4cb7-994f-5338cfbf7579" containerID="9937beacfb2360f1c0f1b804cf3c1f24090939e63648ba77f0f67717d87020ca" exitCode=0
Sep 09 10:48:22.510515 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.510400    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc" event={"ID":"3b1056f9-b34f-4cb7-994f-5338cfbf7579","Type":"ContainerDied","Data":"9937beacfb2360f1c0f1b804cf3c1f24090939e63648ba77f0f67717d87020ca"}
Sep 09 10:48:22.566951 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.566925    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:22.584499 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.584437    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:48:22.670043 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.670009    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:22.751439 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.751408    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-maas-api-tls\") pod \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") "
Sep 09 10:48:22.751939 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.751459    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-metrics-certs\") pod \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") "
Sep 09 10:48:22.751939 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.751501    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1056f9-b34f-4cb7-994f-5338cfbf7579-openshift-service-ca\") pod \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") "
Sep 09 10:48:22.751939 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.751596    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v82d\" (UniqueName: \"kubernetes.io/projected/3b1056f9-b34f-4cb7-994f-5338cfbf7579-kube-api-access-7v82d\") pod \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\" (UID: \"3b1056f9-b34f-4cb7-994f-5338cfbf7579\") "
Sep 09 10:48:22.751939 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.751875    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1056f9-b34f-4cb7-994f-5338cfbf7579-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b1056f9-b34f-4cb7-994f-5338cfbf7579" (UID: "3b1056f9-b34f-4cb7-994f-5338cfbf7579"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:48:22.752159 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.752062    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1056f9-b34f-4cb7-994f-5338cfbf7579-openshift-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:22.753921 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.753891    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1056f9-b34f-4cb7-994f-5338cfbf7579-kube-api-access-7v82d" (OuterVolumeSpecName: "kube-api-access-7v82d") pod "3b1056f9-b34f-4cb7-994f-5338cfbf7579" (UID: "3b1056f9-b34f-4cb7-994f-5338cfbf7579"). InnerVolumeSpecName "kube-api-access-7v82d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:48:22.754177 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.754159    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3b1056f9-b34f-4cb7-994f-5338cfbf7579" (UID: "3b1056f9-b34f-4cb7-994f-5338cfbf7579"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:48:22.754244 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.754221    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b1056f9-b34f-4cb7-994f-5338cfbf7579" (UID: "3b1056f9-b34f-4cb7-994f-5338cfbf7579"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:48:22.853475 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.853444    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v82d\" (UniqueName: \"kubernetes.io/projected/3b1056f9-b34f-4cb7-994f-5338cfbf7579-kube-api-access-7v82d\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:22.853612 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.853491    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-maas-api-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:22.853612 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.853501    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1056f9-b34f-4cb7-994f-5338cfbf7579-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:22.944438 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:22.944404    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:23.516724 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:23.516689    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"
Sep 09 10:48:23.516894 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:23.516725    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc" event={"ID":"3b1056f9-b34f-4cb7-994f-5338cfbf7579","Type":"ContainerDied","Data":"dc8e654686b4307f7b324bcc2b54853413a03cf820f8c624ed5ff07735e2d4a6"}
Sep 09 10:48:23.516894 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:23.516811    2570 scope.go:117] "RemoveContainer" containerID="9937beacfb2360f1c0f1b804cf3c1f24090939e63648ba77f0f67717d87020ca"
Sep 09 10:48:23.533292 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:23.533263    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"]
Sep 09 10:48:23.537705 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:23.537681    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40f1e9-6ffdf644b9-hwfcc"]
Sep 09 10:48:24.839947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:24.839914    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b1056f9-b34f-4cb7-994f-5338cfbf7579" path="/var/lib/kubelet/pods/3b1056f9-b34f-4cb7-994f-5338cfbf7579/volumes"
Sep 09 10:48:25.324160 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.324121    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2" podUID="14f5f1df-cd2f-4c17-b439-ecd6b1915da2" containerName="main" containerID="cri-o://3b1ad11d78411dd37aed74e66f96ab4bcd747593ee950a324dc1b4e6a9829ae3" gracePeriod=45
Sep 09 10:48:25.530356 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.530321    2570 generic.go:356] "Generic (PLEG): container finished" podID="14f5f1df-cd2f-4c17-b439-ecd6b1915da2" containerID="3b1ad11d78411dd37aed74e66f96ab4bcd747593ee950a324dc1b4e6a9829ae3" exitCode=0
Sep 09 10:48:25.530602 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.530406    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2" event={"ID":"14f5f1df-cd2f-4c17-b439-ecd6b1915da2","Type":"ContainerDied","Data":"3b1ad11d78411dd37aed74e66f96ab4bcd747593ee950a324dc1b4e6a9829ae3"}
Sep 09 10:48:25.586931 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.586866    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:48:25.680738 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.680698    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flzqh\" (UniqueName: \"kubernetes.io/projected/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-kube-api-access-flzqh\") pod \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") "
Sep 09 10:48:25.680962 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.680795    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-home\") pod \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") "
Sep 09 10:48:25.680962 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.680844    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-tmp-dir\") pod \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") "
Sep 09 10:48:25.680962 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.680902    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-model-cache\") pod \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") "
Sep 09 10:48:25.680962 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.680919    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-tls-certs\") pod \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") "
Sep 09 10:48:25.680962 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.680947    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-dshm\") pod \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\" (UID: \"14f5f1df-cd2f-4c17-b439-ecd6b1915da2\") "
Sep 09 10:48:25.681241 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.681120    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-home" (OuterVolumeSpecName: "home") pod "14f5f1df-cd2f-4c17-b439-ecd6b1915da2" (UID: "14f5f1df-cd2f-4c17-b439-ecd6b1915da2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:25.681241 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.681177    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-model-cache" (OuterVolumeSpecName: "model-cache") pod "14f5f1df-cd2f-4c17-b439-ecd6b1915da2" (UID: "14f5f1df-cd2f-4c17-b439-ecd6b1915da2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:25.681241 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.681194    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14f5f1df-cd2f-4c17-b439-ecd6b1915da2" (UID: "14f5f1df-cd2f-4c17-b439-ecd6b1915da2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:25.681357 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.681344    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-home\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:25.681392 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.681370    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-tmp-dir\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:25.681392 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.681387    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-model-cache\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:25.683187 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.683153    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-dshm" (OuterVolumeSpecName: "dshm") pod "14f5f1df-cd2f-4c17-b439-ecd6b1915da2" (UID: "14f5f1df-cd2f-4c17-b439-ecd6b1915da2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:25.683293 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.683184    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14f5f1df-cd2f-4c17-b439-ecd6b1915da2" (UID: "14f5f1df-cd2f-4c17-b439-ecd6b1915da2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:48:25.683293 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.683231    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-kube-api-access-flzqh" (OuterVolumeSpecName: "kube-api-access-flzqh") pod "14f5f1df-cd2f-4c17-b439-ecd6b1915da2" (UID: "14f5f1df-cd2f-4c17-b439-ecd6b1915da2"). InnerVolumeSpecName "kube-api-access-flzqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:48:25.782558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.782529    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-tls-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:25.782558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.782555    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-dshm\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:25.782558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:25.782564    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flzqh\" (UniqueName: \"kubernetes.io/projected/14f5f1df-cd2f-4c17-b439-ecd6b1915da2-kube-api-access-flzqh\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:26.537350 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:26.537309    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2" event={"ID":"14f5f1df-cd2f-4c17-b439-ecd6b1915da2","Type":"ContainerDied","Data":"ae3fdb3c40b310bca510fbf4d1b364b54eac85f8abd765d80cdc764bb529c911"}
Sep 09 10:48:26.537350 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:26.537340    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"
Sep 09 10:48:26.537883 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:26.537377    2570 scope.go:117] "RemoveContainer" containerID="3b1ad11d78411dd37aed74e66f96ab4bcd747593ee950a324dc1b4e6a9829ae3"
Sep 09 10:48:26.558818 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:26.558791    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"]
Sep 09 10:48:26.564423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:26.564400    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-40f1e9/test-model-40f1e9-kserve-644b64db75-nqss2"]
Sep 09 10:48:26.839971 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:26.839895    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14f5f1df-cd2f-4c17-b439-ecd6b1915da2" path="/var/lib/kubelet/pods/14f5f1df-cd2f-4c17-b439-ecd6b1915da2/volumes"
Sep 09 10:48:26.864355 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:26.864325    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:27.106460 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.106368    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"]
Sep 09 10:48:27.106638 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.106595    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4" podUID="f7f94307-2cbc-4a82-b1ca-240223286467" containerName="istio-proxy" containerID="cri-o://9642a0af005cb45fdd125c96dba426c0a472ddeabee1f10bb7d618cc7f35c23f" gracePeriod=30
Sep 09 10:48:27.848392 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.848351    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"]
Sep 09 10:48:27.848825 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.848811    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14f5f1df-cd2f-4c17-b439-ecd6b1915da2" containerName="main"
Sep 09 10:48:27.848887 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.848827    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="14f5f1df-cd2f-4c17-b439-ecd6b1915da2" containerName="main"
Sep 09 10:48:27.848887 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.848847    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1056f9-b34f-4cb7-994f-5338cfbf7579" containerName="maas-api"
Sep 09 10:48:27.848887 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.848852    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1056f9-b34f-4cb7-994f-5338cfbf7579" containerName="maas-api"
Sep 09 10:48:27.848887 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.848863    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e194eeed-28db-40df-aead-dfcded573d32" containerName="maas-api"
Sep 09 10:48:27.848887 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.848868    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e194eeed-28db-40df-aead-dfcded573d32" containerName="maas-api"
Sep 09 10:48:27.849066 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.848928    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e194eeed-28db-40df-aead-dfcded573d32" containerName="maas-api"
Sep 09 10:48:27.849066 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.848935    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1056f9-b34f-4cb7-994f-5338cfbf7579" containerName="maas-api"
Sep 09 10:48:27.849066 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.848943    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="14f5f1df-cd2f-4c17-b439-ecd6b1915da2" containerName="main"
Sep 09 10:48:27.853318 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.853295    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:27.856533 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.856506    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-c02105-openshift-default-dockercfg-v5nbw\""
Sep 09 10:48:27.868856 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:27.868832    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"]
Sep 09 10:48:28.005413 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.005370    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxzsd\" (UniqueName: \"kubernetes.io/projected/8ec765a0-ce79-4977-b063-97b8b606d6fe-kube-api-access-nxzsd\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.005594 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.005436    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-workload-certs\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.005594 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.005515    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-credential-socket\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.005594 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.005563    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-podinfo\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.005699 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.005608    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-data\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.005699 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.005670    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-token\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.005799 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.005698    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-envoy\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.005799 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.005737    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-workload-socket\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.005866 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.005829    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec765a0-ce79-4977-b063-97b8b606d6fe-istiod-ca-cert\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.019344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.019319    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:28.079989 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.079944    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:28.107107 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.107019    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzsd\" (UniqueName: \"kubernetes.io/projected/8ec765a0-ce79-4977-b063-97b8b606d6fe-kube-api-access-nxzsd\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.107107 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.107097    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-workload-certs\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.107317 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.107133    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-credential-socket\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.107317 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.107163    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-podinfo\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.107317 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.107197    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-data\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.107317 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.107255    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-token\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.107317 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.107287    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-envoy\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.107571 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.107330    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-workload-socket\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.107571 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.107368    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec765a0-ce79-4977-b063-97b8b606d6fe-istiod-ca-cert\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.107571 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.107563    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-workload-certs\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.107728 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.107638    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-credential-socket\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.107797 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.107734    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-data\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.107866 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.107827    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-workload-socket\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.108082 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.108061    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec765a0-ce79-4977-b063-97b8b606d6fe-istiod-ca-cert\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.109732 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.109707    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-envoy\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.110011 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.109993    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-podinfo\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.115271 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.115248    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-token\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.115673 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.115656    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxzsd\" (UniqueName: \"kubernetes.io/projected/8ec765a0-ce79-4977-b063-97b8b606d6fe-kube-api-access-nxzsd\") pod \"e2e-noten-c02105-openshift-default-75664f979d-875s7\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") " pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.164586 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.164519    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:28.217614 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.217539    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:28.444894 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.444867    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"]
Sep 09 10:48:28.446204 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:48:28.446175    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec765a0_ce79_4977_b063_97b8b606d6fe.slice/crio-272f7ad5be21e20d3bd7d98b4fcc23be05abe9ab00850159b9af2fd255d639a7 WatchSource:0}: Error finding container 272f7ad5be21e20d3bd7d98b4fcc23be05abe9ab00850159b9af2fd255d639a7: Status 404 returned error can't find the container with id 272f7ad5be21e20d3bd7d98b4fcc23be05abe9ab00850159b9af2fd255d639a7
Sep 09 10:48:28.448421 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.448390    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:48:28.448513 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.448466    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:48:28.448582 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.448518    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:48:28.551265 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.551219    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7" event={"ID":"8ec765a0-ce79-4977-b063-97b8b606d6fe","Type":"ContainerStarted","Data":"4bf4581700da1b25f657307ce6b5e7a59e367d2a921ad9713fe33fcec668dff9"}
Sep 09 10:48:28.551424 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.551278    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7" event={"ID":"8ec765a0-ce79-4977-b063-97b8b606d6fe","Type":"ContainerStarted","Data":"272f7ad5be21e20d3bd7d98b4fcc23be05abe9ab00850159b9af2fd255d639a7"}
Sep 09 10:48:28.574648 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.574583    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7" podStartSLOduration=1.574565894 podStartE2EDuration="1.574565894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:48:28.572419365 +0000 UTC m=+1348.236058343" watchObservedRunningTime="2026-09-09 10:48:28.574565894 +0000 UTC m=+1348.238204868"
Sep 09 10:48:28.584999 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.584962    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:28.863922 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.863880    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:28.961883 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:28.961846    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:29.080946 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:29.080911    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:29.165418 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:29.165322    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:29.170337 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:29.170312    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:29.436494 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:29.436415    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:29.533056 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:29.533026    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:29.556566 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:29.556535    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:29.557677 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:29.557651    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:48:29.620010 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:29.619973    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"]
Sep 09 10:48:29.620290 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:29.620259    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy" containerID="cri-o://d5803b8d363d42c16cea2483aa808f6c9102c63165b07a4b00a094f537d0c209" gracePeriod=30
Sep 09 10:48:29.627477 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:29.627449    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:30.138423 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:30.138388    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:32.362223 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.362194    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:48:32.451924 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.451848    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-workload-socket\") pod \"f7f94307-2cbc-4a82-b1ca-240223286467\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") "
Sep 09 10:48:32.451924 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.451894    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-workload-certs\") pod \"f7f94307-2cbc-4a82-b1ca-240223286467\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") "
Sep 09 10:48:32.451924 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.451916    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-istio-envoy\") pod \"f7f94307-2cbc-4a82-b1ca-240223286467\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") "
Sep 09 10:48:32.452196 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.451956    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-credential-socket\") pod \"f7f94307-2cbc-4a82-b1ca-240223286467\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") "
Sep 09 10:48:32.452196 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452004    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f94307-2cbc-4a82-b1ca-240223286467-istio-podinfo\") pod \"f7f94307-2cbc-4a82-b1ca-240223286467\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") "
Sep 09 10:48:32.452196 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452020    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwfc8\" (UniqueName: \"kubernetes.io/projected/f7f94307-2cbc-4a82-b1ca-240223286467-kube-api-access-rwfc8\") pod \"f7f94307-2cbc-4a82-b1ca-240223286467\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") "
Sep 09 10:48:32.452196 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452049    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-istio-data\") pod \"f7f94307-2cbc-4a82-b1ca-240223286467\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") "
Sep 09 10:48:32.452196 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452053    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f7f94307-2cbc-4a82-b1ca-240223286467" (UID: "f7f94307-2cbc-4a82-b1ca-240223286467"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:32.452196 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452100    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f94307-2cbc-4a82-b1ca-240223286467-istiod-ca-cert\") pod \"f7f94307-2cbc-4a82-b1ca-240223286467\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") "
Sep 09 10:48:32.452196 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452153    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f94307-2cbc-4a82-b1ca-240223286467-istio-token\") pod \"f7f94307-2cbc-4a82-b1ca-240223286467\" (UID: \"f7f94307-2cbc-4a82-b1ca-240223286467\") "
Sep 09 10:48:32.452633 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452217    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f7f94307-2cbc-4a82-b1ca-240223286467" (UID: "f7f94307-2cbc-4a82-b1ca-240223286467"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:32.452633 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452310    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f7f94307-2cbc-4a82-b1ca-240223286467" (UID: "f7f94307-2cbc-4a82-b1ca-240223286467"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:32.452633 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452333    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-istio-data" (OuterVolumeSpecName: "istio-data") pod "f7f94307-2cbc-4a82-b1ca-240223286467" (UID: "f7f94307-2cbc-4a82-b1ca-240223286467"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:32.452633 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452510    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-workload-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:32.452633 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452527    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-workload-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:32.452633 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452542    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-credential-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:32.452633 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452560    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-istio-data\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:32.452633 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.452610    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f94307-2cbc-4a82-b1ca-240223286467-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f7f94307-2cbc-4a82-b1ca-240223286467" (UID: "f7f94307-2cbc-4a82-b1ca-240223286467"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:48:32.454207 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.454177    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f7f94307-2cbc-4a82-b1ca-240223286467" (UID: "f7f94307-2cbc-4a82-b1ca-240223286467"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:32.454312 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.454217    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f94307-2cbc-4a82-b1ca-240223286467-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7f94307-2cbc-4a82-b1ca-240223286467" (UID: "f7f94307-2cbc-4a82-b1ca-240223286467"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:48:32.454312 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.454265    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7f94307-2cbc-4a82-b1ca-240223286467-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f7f94307-2cbc-4a82-b1ca-240223286467" (UID: "f7f94307-2cbc-4a82-b1ca-240223286467"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:48:32.454397 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.454307    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f94307-2cbc-4a82-b1ca-240223286467-kube-api-access-rwfc8" (OuterVolumeSpecName: "kube-api-access-rwfc8") pod "f7f94307-2cbc-4a82-b1ca-240223286467" (UID: "f7f94307-2cbc-4a82-b1ca-240223286467"). InnerVolumeSpecName "kube-api-access-rwfc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:48:32.553977 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.553940    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f94307-2cbc-4a82-b1ca-240223286467-istio-podinfo\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:32.553977 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.553970    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwfc8\" (UniqueName: \"kubernetes.io/projected/f7f94307-2cbc-4a82-b1ca-240223286467-kube-api-access-rwfc8\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:32.553977 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.553981    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f94307-2cbc-4a82-b1ca-240223286467-istiod-ca-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:32.553977 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.553990    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f94307-2cbc-4a82-b1ca-240223286467-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:32.554242 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.553998    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f94307-2cbc-4a82-b1ca-240223286467-istio-envoy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:32.573446 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.573411    2570 generic.go:356] "Generic (PLEG): container finished" podID="f7f94307-2cbc-4a82-b1ca-240223286467" containerID="9642a0af005cb45fdd125c96dba426c0a472ddeabee1f10bb7d618cc7f35c23f" exitCode=0
Sep 09 10:48:32.573577 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.573466    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"
Sep 09 10:48:32.573577 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.573475    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4" event={"ID":"f7f94307-2cbc-4a82-b1ca-240223286467","Type":"ContainerDied","Data":"9642a0af005cb45fdd125c96dba426c0a472ddeabee1f10bb7d618cc7f35c23f"}
Sep 09 10:48:32.573577 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.573530    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4" event={"ID":"f7f94307-2cbc-4a82-b1ca-240223286467","Type":"ContainerDied","Data":"656d2d0db3be61d2a0c84ce75aae34efbd10327addaac53583e6b40942f69246"}
Sep 09 10:48:32.573577 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.573555    2570 scope.go:117] "RemoveContainer" containerID="9642a0af005cb45fdd125c96dba426c0a472ddeabee1f10bb7d618cc7f35c23f"
Sep 09 10:48:32.583484 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.583468    2570 scope.go:117] "RemoveContainer" containerID="b931af0997f156c8fb04ff21ae384ca85a2dd56880745ece96c46e039c3ae161"
Sep 09 10:48:32.591274 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.591258    2570 scope.go:117] "RemoveContainer" containerID="9642a0af005cb45fdd125c96dba426c0a472ddeabee1f10bb7d618cc7f35c23f"
Sep 09 10:48:32.591539 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:48:32.591514    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9642a0af005cb45fdd125c96dba426c0a472ddeabee1f10bb7d618cc7f35c23f\": container with ID starting with 9642a0af005cb45fdd125c96dba426c0a472ddeabee1f10bb7d618cc7f35c23f not found: ID does not exist" containerID="9642a0af005cb45fdd125c96dba426c0a472ddeabee1f10bb7d618cc7f35c23f"
Sep 09 10:48:32.591579 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.591549    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9642a0af005cb45fdd125c96dba426c0a472ddeabee1f10bb7d618cc7f35c23f"} err="failed to get container status \"9642a0af005cb45fdd125c96dba426c0a472ddeabee1f10bb7d618cc7f35c23f\": rpc error: code = NotFound desc = could not find container \"9642a0af005cb45fdd125c96dba426c0a472ddeabee1f10bb7d618cc7f35c23f\": container with ID starting with 9642a0af005cb45fdd125c96dba426c0a472ddeabee1f10bb7d618cc7f35c23f not found: ID does not exist"
Sep 09 10:48:32.591579 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.591566    2570 scope.go:117] "RemoveContainer" containerID="b931af0997f156c8fb04ff21ae384ca85a2dd56880745ece96c46e039c3ae161"
Sep 09 10:48:32.591841 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:48:32.591823    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b931af0997f156c8fb04ff21ae384ca85a2dd56880745ece96c46e039c3ae161\": container with ID starting with b931af0997f156c8fb04ff21ae384ca85a2dd56880745ece96c46e039c3ae161 not found: ID does not exist" containerID="b931af0997f156c8fb04ff21ae384ca85a2dd56880745ece96c46e039c3ae161"
Sep 09 10:48:32.591877 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.591849    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b931af0997f156c8fb04ff21ae384ca85a2dd56880745ece96c46e039c3ae161"} err="failed to get container status \"b931af0997f156c8fb04ff21ae384ca85a2dd56880745ece96c46e039c3ae161\": rpc error: code = NotFound desc = could not find container \"b931af0997f156c8fb04ff21ae384ca85a2dd56880745ece96c46e039c3ae161\": container with ID starting with b931af0997f156c8fb04ff21ae384ca85a2dd56880745ece96c46e039c3ae161 not found: ID does not exist"
Sep 09 10:48:32.600249 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.600227    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"]
Sep 09 10:48:32.606389 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.606369    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-928d2a-openshift-default-5674d97d6d-m45g4"]
Sep 09 10:48:32.839599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:32.839565    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7f94307-2cbc-4a82-b1ca-240223286467" path="/var/lib/kubelet/pods/f7f94307-2cbc-4a82-b1ca-240223286467/volumes"
Sep 09 10:48:34.547914 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.547879    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:34.661276 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.661238    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:34.777200 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.777162    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:34.876467 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.876445    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:48:34.974930 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.974900    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istiod-ca-cert\") pod \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") "
Sep 09 10:48:34.975093 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.974948    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-workload-socket\") pod \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") "
Sep 09 10:48:34.975093 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.974968    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-podinfo\") pod \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") "
Sep 09 10:48:34.975093 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975032    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhx82\" (UniqueName: \"kubernetes.io/projected/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-kube-api-access-hhx82\") pod \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") "
Sep 09 10:48:34.975093 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975067    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-data\") pod \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") "
Sep 09 10:48:34.975278 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975106    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-envoy\") pod \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") "
Sep 09 10:48:34.975278 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975132    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-token\") pod \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") "
Sep 09 10:48:34.975278 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975157    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-workload-certs\") pod \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") "
Sep 09 10:48:34.975278 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975203    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-credential-socket\") pod \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\" (UID: \"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a\") "
Sep 09 10:48:34.975278 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975226    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" (UID: "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:34.975525 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975395    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-data" (OuterVolumeSpecName: "istio-data") pod "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" (UID: "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:34.975525 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975469    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" (UID: "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:48:34.975525 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975480    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" (UID: "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:34.975683 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975565    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" (UID: "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:34.975683 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975601    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-data\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:34.975683 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975621    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-workload-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:34.975683 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975634    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istiod-ca-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:34.975683 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.975647    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-workload-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:34.977346 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.977322    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-token" (OuterVolumeSpecName: "istio-token") pod "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" (UID: "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:48:34.977473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.977374    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-kube-api-access-hhx82" (OuterVolumeSpecName: "kube-api-access-hhx82") pod "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" (UID: "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a"). InnerVolumeSpecName "kube-api-access-hhx82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:48:34.977678 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.977665    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" (UID: "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:34.977723 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:34.977680    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" (UID: "e7c3d931-4dfd-4d67-ba44-4f9bcb62465a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:48:35.019782 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.019743    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:35.077113 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.077081    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhx82\" (UniqueName: \"kubernetes.io/projected/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-kube-api-access-hhx82\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:35.077113 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.077108    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-envoy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:35.077113 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.077120    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:35.077340 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.077128    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-credential-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:35.077340 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.077137    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a-istio-podinfo\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:48:35.444801 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.444747    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:35.591168 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.591135    2570 generic.go:356] "Generic (PLEG): container finished" podID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerID="d5803b8d363d42c16cea2483aa808f6c9102c63165b07a4b00a094f537d0c209" exitCode=0
Sep 09 10:48:35.591521 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.591180    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"
Sep 09 10:48:35.591521 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.591222    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" event={"ID":"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a","Type":"ContainerDied","Data":"d5803b8d363d42c16cea2483aa808f6c9102c63165b07a4b00a094f537d0c209"}
Sep 09 10:48:35.591521 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.591276    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w" event={"ID":"e7c3d931-4dfd-4d67-ba44-4f9bcb62465a","Type":"ContainerDied","Data":"81374a4157af5ffde04076ccbb132ec574bc8dd71c184e5e4c1002c2c2a7c3e6"}
Sep 09 10:48:35.591521 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.591298    2570 scope.go:117] "RemoveContainer" containerID="d5803b8d363d42c16cea2483aa808f6c9102c63165b07a4b00a094f537d0c209"
Sep 09 10:48:35.608043 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.608021    2570 scope.go:117] "RemoveContainer" containerID="bae3131a7cb477364193ace3607b175db328438246be745af97954abe173e5c5"
Sep 09 10:48:35.616404 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.616380    2570 scope.go:117] "RemoveContainer" containerID="d5803b8d363d42c16cea2483aa808f6c9102c63165b07a4b00a094f537d0c209"
Sep 09 10:48:35.616692 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:48:35.616670    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5803b8d363d42c16cea2483aa808f6c9102c63165b07a4b00a094f537d0c209\": container with ID starting with d5803b8d363d42c16cea2483aa808f6c9102c63165b07a4b00a094f537d0c209 not found: ID does not exist" containerID="d5803b8d363d42c16cea2483aa808f6c9102c63165b07a4b00a094f537d0c209"
Sep 09 10:48:35.616751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.616701    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5803b8d363d42c16cea2483aa808f6c9102c63165b07a4b00a094f537d0c209"} err="failed to get container status \"d5803b8d363d42c16cea2483aa808f6c9102c63165b07a4b00a094f537d0c209\": rpc error: code = NotFound desc = could not find container \"d5803b8d363d42c16cea2483aa808f6c9102c63165b07a4b00a094f537d0c209\": container with ID starting with d5803b8d363d42c16cea2483aa808f6c9102c63165b07a4b00a094f537d0c209 not found: ID does not exist"
Sep 09 10:48:35.616751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.616719    2570 scope.go:117] "RemoveContainer" containerID="bae3131a7cb477364193ace3607b175db328438246be745af97954abe173e5c5"
Sep 09 10:48:35.617077 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:48:35.617060    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae3131a7cb477364193ace3607b175db328438246be745af97954abe173e5c5\": container with ID starting with bae3131a7cb477364193ace3607b175db328438246be745af97954abe173e5c5 not found: ID does not exist" containerID="bae3131a7cb477364193ace3607b175db328438246be745af97954abe173e5c5"
Sep 09 10:48:35.617125 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.617083    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae3131a7cb477364193ace3607b175db328438246be745af97954abe173e5c5"} err="failed to get container status \"bae3131a7cb477364193ace3607b175db328438246be745af97954abe173e5c5\": rpc error: code = NotFound desc = could not find container \"bae3131a7cb477364193ace3607b175db328438246be745af97954abe173e5c5\": container with ID starting with bae3131a7cb477364193ace3607b175db328438246be745af97954abe173e5c5 not found: ID does not exist"
Sep 09 10:48:35.617881 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.617864    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"]
Sep 09 10:48:35.622350 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.622329    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-40f1e9-openshift-default-6dc777c58d-lxc8w"]
Sep 09 10:48:35.771638 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.771600    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:35.873940 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:35.873900    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:36.839863 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:36.839828    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" path="/var/lib/kubelet/pods/e7c3d931-4dfd-4d67-ba44-4f9bcb62465a/volumes"
Sep 09 10:48:37.158183 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:37.158093    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:38.688191 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.688160    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"]
Sep 09 10:48:38.688595 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.688529    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy"
Sep 09 10:48:38.688595 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.688540    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy"
Sep 09 10:48:38.688595 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.688547    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f94307-2cbc-4a82-b1ca-240223286467" containerName="istio-proxy"
Sep 09 10:48:38.688595 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.688552    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f94307-2cbc-4a82-b1ca-240223286467" containerName="istio-proxy"
Sep 09 10:48:38.688595 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.688565    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f94307-2cbc-4a82-b1ca-240223286467" containerName="istio-proxy"
Sep 09 10:48:38.688595 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.688570    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f94307-2cbc-4a82-b1ca-240223286467" containerName="istio-proxy"
Sep 09 10:48:38.688595 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.688583    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy"
Sep 09 10:48:38.688595 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.688589    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy"
Sep 09 10:48:38.688869 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.688642    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f94307-2cbc-4a82-b1ca-240223286467" containerName="istio-proxy"
Sep 09 10:48:38.688869 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.688650    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f94307-2cbc-4a82-b1ca-240223286467" containerName="istio-proxy"
Sep 09 10:48:38.688869 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.688656    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy"
Sep 09 10:48:38.688869 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.688663    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c3d931-4dfd-4d67-ba44-4f9bcb62465a" containerName="istio-proxy"
Sep 09 10:48:38.693438 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.693417    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.696336 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.696311    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c02105\"/\"openshift-service-ca.crt\""
Sep 09 10:48:38.697704 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.697675    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c02105\"/\"default-dockercfg-zdbz8\""
Sep 09 10:48:38.697834 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.697682    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c02105\"/\"noten-model-c02105-kserve-self-signed-certs\""
Sep 09 10:48:38.697834 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.697683    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c02105\"/\"kube-root-ca.crt\""
Sep 09 10:48:38.703804 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.703752    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"]
Sep 09 10:48:38.726352 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.726299    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:38.812622 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.812587    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj288\" (UniqueName: \"kubernetes.io/projected/63b7dec3-984a-419b-a976-f64998bbdfaf-kube-api-access-dj288\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.812823 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.812629    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-model-cache\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.812823 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.812651    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b7dec3-984a-419b-a976-f64998bbdfaf-tls-certs\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.812823 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.812729    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-tmp-dir\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.812823 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.812814    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-dshm\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.813025 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.812868    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-home\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.914266 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.914223    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj288\" (UniqueName: \"kubernetes.io/projected/63b7dec3-984a-419b-a976-f64998bbdfaf-kube-api-access-dj288\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.914472 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.914276    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-model-cache\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.914472 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.914302    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b7dec3-984a-419b-a976-f64998bbdfaf-tls-certs\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.914472 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.914325    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-tmp-dir\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.914472 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.914343    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-dshm\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.914472 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.914425    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-home\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.914835 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.914811    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-tmp-dir\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.914909 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.914823    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-model-cache\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.914909 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.914879    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-home\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.916742 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.916720    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-dshm\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.917146 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.917124    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b7dec3-984a-419b-a976-f64998bbdfaf-tls-certs\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:38.922536 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:38.922506    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj288\" (UniqueName: \"kubernetes.io/projected/63b7dec3-984a-419b-a976-f64998bbdfaf-kube-api-access-dj288\") pod \"noten-model-c02105-kserve-79995cc547-cnkvc\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") " pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:39.003726 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:39.003693    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:39.277375 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:39.277340    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"]
Sep 09 10:48:39.279412 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:48:39.279383    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63b7dec3_984a_419b_a976_f64998bbdfaf.slice/crio-7a946f8f2a51375b5217497a9fe975d5d3d863cd9a0e156891f8372d1a0f47d5 WatchSource:0}: Error finding container 7a946f8f2a51375b5217497a9fe975d5d3d863cd9a0e156891f8372d1a0f47d5: Status 404 returned error can't find the container with id 7a946f8f2a51375b5217497a9fe975d5d3d863cd9a0e156891f8372d1a0f47d5
Sep 09 10:48:39.617920 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:39.617827    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc" event={"ID":"63b7dec3-984a-419b-a976-f64998bbdfaf","Type":"ContainerStarted","Data":"3f506dd7089f27264ad54fe1e17e62aed83c40f177e1b9b8b97d99739a83fa4b"}
Sep 09 10:48:39.617920 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:39.617871    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc" event={"ID":"63b7dec3-984a-419b-a976-f64998bbdfaf","Type":"ContainerStarted","Data":"7a946f8f2a51375b5217497a9fe975d5d3d863cd9a0e156891f8372d1a0f47d5"}
Sep 09 10:48:39.636231 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:39.636177    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc" podStartSLOduration=1.636159798 podStartE2EDuration="1.636159798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:48:39.633687521 +0000 UTC m=+1359.297326495" watchObservedRunningTime="2026-09-09 10:48:39.636159798 +0000 UTC m=+1359.299798822"
Sep 09 10:48:40.582246 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:40.582206    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:40.650297 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:40.650253    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:40.827059 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:40.827010    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:40.896852 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:40.896752    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:41.120556 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:41.120494    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:43.372059 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:43.371997    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:43.442922 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:43.442876    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:44.037352 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:44.037318    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:47.457170 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:47.457131    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:47.563286 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:47.563222    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:47.717476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:47.717389    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:48.283724 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.283692    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:48.567228 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.567141    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"]
Sep 09 10:48:48.571430 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.571403    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.574150 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.574125    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1eb21d-openshift-default-dockercfg-qh6j9\""
Sep 09 10:48:48.582339 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.582310    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"]
Sep 09 10:48:48.596949 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.596922    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:48.712776 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.712732    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:48.715642 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.715611    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-token\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.715642 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.715643    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7v7w\" (UniqueName: \"kubernetes.io/projected/1958cc9b-9cd5-4f98-b693-607d925c4be8-kube-api-access-b7v7w\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.715856 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.715666    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-data\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.715856 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.715820    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-workload-certs\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.715856 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.715854    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-workload-socket\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.715994 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.715879    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1958cc9b-9cd5-4f98-b693-607d925c4be8-istiod-ca-cert\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.715994 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.715897    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-credential-socket\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.715994 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.715916    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-podinfo\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.716135 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.716018    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-envoy\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817014 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.816977    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-workload-certs\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817014 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.817018    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-workload-socket\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817310 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.817047    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1958cc9b-9cd5-4f98-b693-607d925c4be8-istiod-ca-cert\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817310 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.817071    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-credential-socket\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817310 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.817087    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-podinfo\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817310 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.817134    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-envoy\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817310 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.817196    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-token\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817310 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.817226    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7v7w\" (UniqueName: \"kubernetes.io/projected/1958cc9b-9cd5-4f98-b693-607d925c4be8-kube-api-access-b7v7w\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817310 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.817262    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-data\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.817487    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-workload-socket\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.817520    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-workload-certs\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.817553    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-credential-socket\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.817668    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-data\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.817915 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.817898    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1958cc9b-9cd5-4f98-b693-607d925c4be8-istiod-ca-cert\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.819548 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.819525    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-envoy\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.819748 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.819726    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-podinfo\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.826967 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.826939    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7v7w\" (UniqueName: \"kubernetes.io/projected/1958cc9b-9cd5-4f98-b693-607d925c4be8-kube-api-access-b7v7w\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.827095 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.827080    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-token\") pod \"e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") " pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:48.883627 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:48.883591    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:49.004656 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:49.004626    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:49.004920 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:49.004679    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:49.017156 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:49.017128    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:49.168371 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:49.168346    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"]
Sep 09 10:48:49.170216 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:48:49.170191    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1958cc9b_9cd5_4f98_b693_607d925c4be8.slice/crio-28935687756d6c3f066f1fdf9226781f8c7ac18d13291c48879a476025a1b36a WatchSource:0}: Error finding container 28935687756d6c3f066f1fdf9226781f8c7ac18d13291c48879a476025a1b36a: Status 404 returned error can't find the container with id 28935687756d6c3f066f1fdf9226781f8c7ac18d13291c48879a476025a1b36a
Sep 09 10:48:49.172862 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:49.172834    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:48:49.172933 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:49.172906    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:48:49.172970 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:49.172937    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:48:49.673071 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:49.673032    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" event={"ID":"1958cc9b-9cd5-4f98-b693-607d925c4be8","Type":"ContainerStarted","Data":"51c2aa0f949ef25d8b17121ff6ad21599f577a266b9f57daa6dde269ff9fdc62"}
Sep 09 10:48:49.673566 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:49.673086    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" event={"ID":"1958cc9b-9cd5-4f98-b693-607d925c4be8","Type":"ContainerStarted","Data":"28935687756d6c3f066f1fdf9226781f8c7ac18d13291c48879a476025a1b36a"}
Sep 09 10:48:49.684495 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:49.684459    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:48:49.695373 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:49.695319    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" podStartSLOduration=1.695303452 podStartE2EDuration="1.695303452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:48:49.693946334 +0000 UTC m=+1369.357585309" watchObservedRunningTime="2026-09-09 10:48:49.695303452 +0000 UTC m=+1369.358942426"
Sep 09 10:48:49.884513 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:49.884481    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:50.600902 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:50.600866    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:50.736215 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:50.736179    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:50.860614 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:50.860528    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:50.890081 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:50.890053    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:51.160331 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:51.160243    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:51.405915 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:51.405878    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:51.683524 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:51.683492    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:51.684620 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:51.684602    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:48:51.740329 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:51.740293    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:51.826164 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:51.826125    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:53.048899 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:53.048868    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:53.149005 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:53.148964    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:53.275130 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:53.275099    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:53.628672 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:53.628641    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:53.736070 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:53.736041    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:54.396128 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:54.396093    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:54.566364 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:54.566329    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:55.608057 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.608024    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"]
Sep 09 10:48:55.612022 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.611990    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:55.614967 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.614940    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2a090f42-serving-cert\""
Sep 09 10:48:55.624973 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.624945    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"]
Sep 09 10:48:55.645134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.645081    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:55.688316 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.688274    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b31e44a-a347-46b3-9399-bce12c382a67-openshift-service-ca\") pod \"maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:55.688494 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.688352    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhc8g\" (UniqueName: \"kubernetes.io/projected/5b31e44a-a347-46b3-9399-bce12c382a67-kube-api-access-qhc8g\") pod \"maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:55.688494 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.688395    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-metrics-certs\") pod \"maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:55.688494 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.688429    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-maas-api-tls\") pod \"maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:55.746055 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.745979    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:55.789334 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.789299    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhc8g\" (UniqueName: \"kubernetes.io/projected/5b31e44a-a347-46b3-9399-bce12c382a67-kube-api-access-qhc8g\") pod \"maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:55.789334 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.789337    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-metrics-certs\") pod \"maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:55.789585 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.789359    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-maas-api-tls\") pod \"maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:55.789585 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.789416    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b31e44a-a347-46b3-9399-bce12c382a67-openshift-service-ca\") pod \"maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:55.789585 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:48:55.789522    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-serving-cert: secret "maas-api-e2e-mt-2a090f42-serving-cert" not found
Sep 09 10:48:55.789735 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:48:55.789604    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-maas-api-tls podName:5b31e44a-a347-46b3-9399-bce12c382a67 nodeName:}" failed. No retries permitted until 2026-09-09 10:48:56.289584264 +0000 UTC m=+1375.953223222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-maas-api-tls") pod "maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm" (UID: "5b31e44a-a347-46b3-9399-bce12c382a67") : secret "maas-api-e2e-mt-2a090f42-serving-cert" not found
Sep 09 10:48:55.790066 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.790045    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b31e44a-a347-46b3-9399-bce12c382a67-openshift-service-ca\") pod \"maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:55.791958 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.791926    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-metrics-certs\") pod \"maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:55.800477 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.800449    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhc8g\" (UniqueName: \"kubernetes.io/projected/5b31e44a-a347-46b3-9399-bce12c382a67-kube-api-access-qhc8g\") pod \"maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:55.994589 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:55.994555    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:56.148409 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:56.148375    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:56.253593 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:56.253503    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:48:56.294238 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:56.294197    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-maas-api-tls\") pod \"maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:56.296887 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:56.296857    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-maas-api-tls\") pod \"maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:56.524803 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:56.524700    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:56.797921 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:56.797901    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"]
Sep 09 10:48:56.800292 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:48:56.800267    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b31e44a_a347_46b3_9399_bce12c382a67.slice/crio-0b7edcedc3dc649a9820418f65635fbf5e103d47621739652535a7c9dc5aa0db WatchSource:0}: Error finding container 0b7edcedc3dc649a9820418f65635fbf5e103d47621739652535a7c9dc5aa0db: Status 404 returned error can't find the container with id 0b7edcedc3dc649a9820418f65635fbf5e103d47621739652535a7c9dc5aa0db
Sep 09 10:48:56.801985 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:56.801968    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 10:48:57.718139 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:57.718101    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm" event={"ID":"5b31e44a-a347-46b3-9399-bce12c382a67","Type":"ContainerStarted","Data":"21f8d8160bd773d89836012adc331bb05b389c7076d285473f9dae1b4a768a26"}
Sep 09 10:48:57.718369 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:57.718157    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm" event={"ID":"5b31e44a-a347-46b3-9399-bce12c382a67","Type":"ContainerStarted","Data":"0b7edcedc3dc649a9820418f65635fbf5e103d47621739652535a7c9dc5aa0db"}
Sep 09 10:48:57.718369 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:57.718233    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:48:57.737566 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:57.737511    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm" podStartSLOduration=2.349729846 podStartE2EDuration="2.737491687s" totalImagesPullingTime="387.761841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:48:57.735829183 +0000 UTC m=+1377.399468157" watchObservedRunningTime="2026-09-09 10:48:57.737491687 +0000 UTC m=+1377.401130660"
Sep 09 10:48:59.393079 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:48:59.392983    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:03.729127 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:03.729097    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:49:07.132001 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:07.131958    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": read tcp 10.134.0.2:52872->10.134.0.61:15021: read: connection reset by peer" start-of-body=
Sep 09 10:49:07.132428 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:07.132047    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": read tcp 10.134.0.2:52872->10.134.0.61:15021: read: connection reset by peer"
Sep 09 10:49:07.800395 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:07.800363    2570 generic.go:356] "Generic (PLEG): container finished" podID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerID="51c2aa0f949ef25d8b17121ff6ad21599f577a266b9f57daa6dde269ff9fdc62" exitCode=0
Sep 09 10:49:07.800579 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:07.800436    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" event={"ID":"1958cc9b-9cd5-4f98-b693-607d925c4be8","Type":"ContainerDied","Data":"51c2aa0f949ef25d8b17121ff6ad21599f577a266b9f57daa6dde269ff9fdc62"}
Sep 09 10:49:07.800875 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:07.800860    2570 scope.go:117] "RemoveContainer" containerID="51c2aa0f949ef25d8b17121ff6ad21599f577a266b9f57daa6dde269ff9fdc62"
Sep 09 10:49:07.801957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:07.801930    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:49:07.802020 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:07.801998    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:49:07.802056 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:07.802030    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:49:08.807610 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:08.807573    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" event={"ID":"1958cc9b-9cd5-4f98-b693-607d925c4be8","Type":"ContainerStarted","Data":"a1502bb8307bb942cc3b8fb6b374413f28d5b8f07682eb6746393512c0d09121"}
Sep 09 10:49:08.885137 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:08.885104    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:49:08.885510 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:08.885424    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 10:49:08.885684 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:08.885659    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 10:49:09.884217 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:09.884176    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 10:49:09.884619 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:09.884240    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 10:49:10.405151 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:10.405120    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"]
Sep 09 10:49:10.405384 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:10.405358    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7" podUID="8ec765a0-ce79-4977-b063-97b8b606d6fe" containerName="istio-proxy" containerID="cri-o://4bf4581700da1b25f657307ce6b5e7a59e367d2a921ad9713fe33fcec668dff9" gracePeriod=30
Sep 09 10:49:10.435291 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:10.435257    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:10.631283 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:10.631250    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:10.765294 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:10.765251    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:10.884126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:10.884091    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 10:49:10.884307 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:10.884147    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 10:49:11.884521 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:11.884487    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 10:49:11.884935 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:11.884566    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 10:49:12.884118 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:12.884082    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 10:49:12.884293 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:12.884152    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 10:49:13.884751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:13.884714    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 10:49:13.885161 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:13.884791    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 10:49:14.050922 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:14.050879    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:14.559035 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:14.558988    2570 patch_prober.go:28] interesting pod/e2e-noten-c02105-openshift-default-75664f979d-875s7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 10:49:14.559214 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:14.559052    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7" podUID="8ec765a0-ce79-4977-b063-97b8b606d6fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 10:49:14.888021 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:14.887935    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:49:14.888400 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:14.888237    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:49:14.889013 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:14.888995    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 10:49:15.660916 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.660893    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:49:15.713536 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.713432    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-podinfo\") pod \"8ec765a0-ce79-4977-b063-97b8b606d6fe\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") "
Sep 09 10:49:15.713536 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.713502    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-envoy\") pod \"8ec765a0-ce79-4977-b063-97b8b606d6fe\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") "
Sep 09 10:49:15.713536 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.713533    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-workload-socket\") pod \"8ec765a0-ce79-4977-b063-97b8b606d6fe\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") "
Sep 09 10:49:15.714055 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.713582    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-workload-certs\") pod \"8ec765a0-ce79-4977-b063-97b8b606d6fe\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") "
Sep 09 10:49:15.714055 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.713607    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-credential-socket\") pod \"8ec765a0-ce79-4977-b063-97b8b606d6fe\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") "
Sep 09 10:49:15.714055 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.713640    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxzsd\" (UniqueName: \"kubernetes.io/projected/8ec765a0-ce79-4977-b063-97b8b606d6fe-kube-api-access-nxzsd\") pod \"8ec765a0-ce79-4977-b063-97b8b606d6fe\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") "
Sep 09 10:49:15.714055 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.713709    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec765a0-ce79-4977-b063-97b8b606d6fe-istiod-ca-cert\") pod \"8ec765a0-ce79-4977-b063-97b8b606d6fe\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") "
Sep 09 10:49:15.714055 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.713734    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-data\") pod \"8ec765a0-ce79-4977-b063-97b8b606d6fe\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") "
Sep 09 10:49:15.714055 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.713783    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-token\") pod \"8ec765a0-ce79-4977-b063-97b8b606d6fe\" (UID: \"8ec765a0-ce79-4977-b063-97b8b606d6fe\") "
Sep 09 10:49:15.714055 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.713933    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8ec765a0-ce79-4977-b063-97b8b606d6fe" (UID: "8ec765a0-ce79-4977-b063-97b8b606d6fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:15.714055 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.713943    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8ec765a0-ce79-4977-b063-97b8b606d6fe" (UID: "8ec765a0-ce79-4977-b063-97b8b606d6fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:15.714736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.714145    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8ec765a0-ce79-4977-b063-97b8b606d6fe" (UID: "8ec765a0-ce79-4977-b063-97b8b606d6fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:15.714736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.714213    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-workload-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:15.714736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.714227    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-workload-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:15.714736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.714228    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec765a0-ce79-4977-b063-97b8b606d6fe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8ec765a0-ce79-4977-b063-97b8b606d6fe" (UID: "8ec765a0-ce79-4977-b063-97b8b606d6fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:49:15.714736 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.714443    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-data" (OuterVolumeSpecName: "istio-data") pod "8ec765a0-ce79-4977-b063-97b8b606d6fe" (UID: "8ec765a0-ce79-4977-b063-97b8b606d6fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:15.715953 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.715932    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8ec765a0-ce79-4977-b063-97b8b606d6fe" (UID: "8ec765a0-ce79-4977-b063-97b8b606d6fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:49:15.716065 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.716046    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8ec765a0-ce79-4977-b063-97b8b606d6fe" (UID: "8ec765a0-ce79-4977-b063-97b8b606d6fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:15.716668 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.716650    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-token" (OuterVolumeSpecName: "istio-token") pod "8ec765a0-ce79-4977-b063-97b8b606d6fe" (UID: "8ec765a0-ce79-4977-b063-97b8b606d6fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:49:15.716728 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.716662    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec765a0-ce79-4977-b063-97b8b606d6fe-kube-api-access-nxzsd" (OuterVolumeSpecName: "kube-api-access-nxzsd") pod "8ec765a0-ce79-4977-b063-97b8b606d6fe" (UID: "8ec765a0-ce79-4977-b063-97b8b606d6fe"). InnerVolumeSpecName "kube-api-access-nxzsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:49:15.747032 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.747004    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"]
Sep 09 10:49:15.799699 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.799667    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:15.815028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.814993    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-envoy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:15.815028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.815024    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-credential-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:15.815214 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.815037    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxzsd\" (UniqueName: \"kubernetes.io/projected/8ec765a0-ce79-4977-b063-97b8b606d6fe-kube-api-access-nxzsd\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:15.815214 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.815049    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec765a0-ce79-4977-b063-97b8b606d6fe-istiod-ca-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:15.815214 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.815061    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-data\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:15.815214 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.815071    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:15.815214 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.815081    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec765a0-ce79-4977-b063-97b8b606d6fe-istio-podinfo\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:15.847304 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.847271    2570 generic.go:356] "Generic (PLEG): container finished" podID="8ec765a0-ce79-4977-b063-97b8b606d6fe" containerID="4bf4581700da1b25f657307ce6b5e7a59e367d2a921ad9713fe33fcec668dff9" exitCode=0
Sep 09 10:49:15.847457 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.847318    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"
Sep 09 10:49:15.847457 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.847361    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7" event={"ID":"8ec765a0-ce79-4977-b063-97b8b606d6fe","Type":"ContainerDied","Data":"4bf4581700da1b25f657307ce6b5e7a59e367d2a921ad9713fe33fcec668dff9"}
Sep 09 10:49:15.847457 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.847418    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7" event={"ID":"8ec765a0-ce79-4977-b063-97b8b606d6fe","Type":"ContainerDied","Data":"272f7ad5be21e20d3bd7d98b4fcc23be05abe9ab00850159b9af2fd255d639a7"}
Sep 09 10:49:15.847457 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.847447    2570 scope.go:117] "RemoveContainer" containerID="4bf4581700da1b25f657307ce6b5e7a59e367d2a921ad9713fe33fcec668dff9"
Sep 09 10:49:15.857402 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.857381    2570 scope.go:117] "RemoveContainer" containerID="4bf4581700da1b25f657307ce6b5e7a59e367d2a921ad9713fe33fcec668dff9"
Sep 09 10:49:15.857691 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:49:15.857671    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf4581700da1b25f657307ce6b5e7a59e367d2a921ad9713fe33fcec668dff9\": container with ID starting with 4bf4581700da1b25f657307ce6b5e7a59e367d2a921ad9713fe33fcec668dff9 not found: ID does not exist" containerID="4bf4581700da1b25f657307ce6b5e7a59e367d2a921ad9713fe33fcec668dff9"
Sep 09 10:49:15.857741 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.857699    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf4581700da1b25f657307ce6b5e7a59e367d2a921ad9713fe33fcec668dff9"} err="failed to get container status \"4bf4581700da1b25f657307ce6b5e7a59e367d2a921ad9713fe33fcec668dff9\": rpc error: code = NotFound desc = could not find container \"4bf4581700da1b25f657307ce6b5e7a59e367d2a921ad9713fe33fcec668dff9\": container with ID starting with 4bf4581700da1b25f657307ce6b5e7a59e367d2a921ad9713fe33fcec668dff9 not found: ID does not exist"
Sep 09 10:49:15.870563 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.870539    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"]
Sep 09 10:49:15.874700 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:15.874678    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-c02105-openshift-default-75664f979d-875s7"]
Sep 09 10:49:16.619862 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:16.619823    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:16.720825 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:16.720760    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:16.839473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:16.839438    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ec765a0-ce79-4977-b063-97b8b606d6fe" path="/var/lib/kubelet/pods/8ec765a0-ce79-4977-b063-97b8b606d6fe/volumes"
Sep 09 10:49:16.966119 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:16.966030    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:17.311124 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.311093    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"]
Sep 09 10:49:17.311580 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.311560    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec765a0-ce79-4977-b063-97b8b606d6fe" containerName="istio-proxy"
Sep 09 10:49:17.311647 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.311582    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec765a0-ce79-4977-b063-97b8b606d6fe" containerName="istio-proxy"
Sep 09 10:49:17.311686 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.311667    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec765a0-ce79-4977-b063-97b8b606d6fe" containerName="istio-proxy"
Sep 09 10:49:17.314439 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.314420    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.326135 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.326113    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-535b75-openshift-default-dockercfg-tw96n\""
Sep 09 10:49:17.339560 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.339533    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"]
Sep 09 10:49:17.382342 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.382302    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:17.432572 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.432542    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-envoy\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.432731 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.432581    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c391eaca-317f-4de8-994b-f6d246ae8f9b-istiod-ca-cert\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.432731 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.432603    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-token\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.432731 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.432630    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-podinfo\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.432731 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.432684    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-credential-socket\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.432891 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.432731    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-data\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.432891 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.432793    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-workload-socket\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.432891 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.432824    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-workload-certs\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.432981 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.432904    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-574dt\" (UniqueName: \"kubernetes.io/projected/c391eaca-317f-4de8-994b-f6d246ae8f9b-kube-api-access-574dt\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.533951 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.533908    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-envoy\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.533951 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.533958    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c391eaca-317f-4de8-994b-f6d246ae8f9b-istiod-ca-cert\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.534205 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.533978    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-token\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.534205 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.533995    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-podinfo\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.534205 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.534019    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-credential-socket\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.534205 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.534047    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-data\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.534205 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.534087    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-workload-socket\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.534205 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.534128    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-workload-certs\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.534205 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.534188    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-574dt\" (UniqueName: \"kubernetes.io/projected/c391eaca-317f-4de8-994b-f6d246ae8f9b-kube-api-access-574dt\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.534556 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.534466    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-data\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.534556 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.534479    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-credential-socket\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.534665 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.534602    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-workload-certs\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.534778 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.534736    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-workload-socket\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.535205 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.535186    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c391eaca-317f-4de8-994b-f6d246ae8f9b-istiod-ca-cert\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.536386 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.536366    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-envoy\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.536717 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.536700    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-podinfo\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.542643 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.542620    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-token\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.542991 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.542972    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-574dt\" (UniqueName: \"kubernetes.io/projected/c391eaca-317f-4de8-994b-f6d246ae8f9b-kube-api-access-574dt\") pod \"e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") " pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.624437 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.624348    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:17.667019 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.666981    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:17.903507 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.903481    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"]
Sep 09 10:49:17.915374 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.915343    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:49:17.915448 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.915418    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:49:17.915491 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:17.915464    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:49:18.864848 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:18.864806    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" event={"ID":"c391eaca-317f-4de8-994b-f6d246ae8f9b","Type":"ContainerStarted","Data":"95b966dc1777b7745de817054f8ce1f16881b2cfefdd281ede0062be3ef3b844"}
Sep 09 10:49:18.865462 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:18.864878    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" event={"ID":"c391eaca-317f-4de8-994b-f6d246ae8f9b","Type":"ContainerStarted","Data":"c1670a6ea6276648a3abf5094f2a667e9ef3a0212ac41e1793e5dc464034c977"}
Sep 09 10:49:18.909527 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:18.909467    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" podStartSLOduration=1.909450449 podStartE2EDuration="1.909450449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:49:18.907475966 +0000 UTC m=+1398.571114940" watchObservedRunningTime="2026-09-09 10:49:18.909450449 +0000 UTC m=+1398.573089423"
Sep 09 10:49:19.208967 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:19.208877    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:19.509249 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:19.509219    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:19.624593 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:19.624559    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:19.629363 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:19.629331    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:19.870000 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:19.869913    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:19.871084 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:19.871062    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:19.889242 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:19.889217    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:19.985634 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:19.985602    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:20.309573 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:20.309535    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:20.903203 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:20.903171    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:20.998889 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:20.998853    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:22.027590 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.027554    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"]
Sep 09 10:49:22.031353 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.031334    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.033732 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.033711    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-535b75-serving-cert\""
Sep 09 10:49:22.042545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.042518    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"]
Sep 09 10:49:22.065493 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.065470    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:22.165890 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.165861    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:22.179244 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.179216    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14e53582-dca0-4305-a883-a2d2e23eaa8f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.179315 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.179249    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e53582-dca0-4305-a883-a2d2e23eaa8f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.179315 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.179281    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e53582-dca0-4305-a883-a2d2e23eaa8f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.179422 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.179404    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn5lj\" (UniqueName: \"kubernetes.io/projected/14e53582-dca0-4305-a883-a2d2e23eaa8f-kube-api-access-qn5lj\") pod \"maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.223950 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.223913    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:22.280599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.280502    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5lj\" (UniqueName: \"kubernetes.io/projected/14e53582-dca0-4305-a883-a2d2e23eaa8f-kube-api-access-qn5lj\") pod \"maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.280599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.280582    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14e53582-dca0-4305-a883-a2d2e23eaa8f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.280872 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.280607    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e53582-dca0-4305-a883-a2d2e23eaa8f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.280872 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.280639    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e53582-dca0-4305-a883-a2d2e23eaa8f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.281419 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.281398    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e53582-dca0-4305-a883-a2d2e23eaa8f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.283142 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.283120    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e53582-dca0-4305-a883-a2d2e23eaa8f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.283244 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.283157    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14e53582-dca0-4305-a883-a2d2e23eaa8f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.289164 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.289140    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5lj\" (UniqueName: \"kubernetes.io/projected/14e53582-dca0-4305-a883-a2d2e23eaa8f-kube-api-access-qn5lj\") pod \"maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.343507 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.343471    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:22.616627 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.616598    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"]
Sep 09 10:49:22.618734 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:49:22.618705    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e53582_dca0_4305_a883_a2d2e23eaa8f.slice/crio-0d47e0300d66d152abf5c3cc86dae3adb15f2bee24eb9bcb5329e61e80a29b6e WatchSource:0}: Error finding container 0d47e0300d66d152abf5c3cc86dae3adb15f2bee24eb9bcb5329e61e80a29b6e: Status 404 returned error can't find the container with id 0d47e0300d66d152abf5c3cc86dae3adb15f2bee24eb9bcb5329e61e80a29b6e
Sep 09 10:49:22.652966 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.652898    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:22.892508 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:22.892408    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d" event={"ID":"14e53582-dca0-4305-a883-a2d2e23eaa8f","Type":"ContainerStarted","Data":"0d47e0300d66d152abf5c3cc86dae3adb15f2bee24eb9bcb5329e61e80a29b6e"}
Sep 09 10:49:23.898992 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:23.898959    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d" event={"ID":"14e53582-dca0-4305-a883-a2d2e23eaa8f","Type":"ContainerStarted","Data":"a357004904d9da55e1e99c2b0fd6141fc2204f377808ccee7ba110c96e6dae4c"}
Sep 09 10:49:23.899414 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:23.899048    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:23.916407 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:23.916359    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d" podStartSLOduration=1.566806345 podStartE2EDuration="1.916345464s" totalImagesPullingTime="349.539119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:49:23.914908239 +0000 UTC m=+1403.578547213" watchObservedRunningTime="2026-09-09 10:49:23.916345464 +0000 UTC m=+1403.579984438"
Sep 09 10:49:24.095349 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:24.095315    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:26.428144 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:26.428108    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:26.529040 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:26.529006    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:26.908816 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:26.908782    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:29.909417 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:29.909389    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 10:49:30.766974 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:30.766933    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc" podUID="63b7dec3-984a-419b-a976-f64998bbdfaf" containerName="main" containerID="cri-o://3f506dd7089f27264ad54fe1e17e62aed83c40f177e1b9b8b97d99739a83fa4b" gracePeriod=45
Sep 09 10:49:30.936075 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:30.935727    2570 generic.go:356] "Generic (PLEG): container finished" podID="63b7dec3-984a-419b-a976-f64998bbdfaf" containerID="3f506dd7089f27264ad54fe1e17e62aed83c40f177e1b9b8b97d99739a83fa4b" exitCode=0
Sep 09 10:49:30.936075 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:30.935950    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc" event={"ID":"63b7dec3-984a-419b-a976-f64998bbdfaf","Type":"ContainerDied","Data":"3f506dd7089f27264ad54fe1e17e62aed83c40f177e1b9b8b97d99739a83fa4b"}
Sep 09 10:49:31.037470 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.037446    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:49:31.177242 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.177183    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj288\" (UniqueName: \"kubernetes.io/projected/63b7dec3-984a-419b-a976-f64998bbdfaf-kube-api-access-dj288\") pod \"63b7dec3-984a-419b-a976-f64998bbdfaf\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") "
Sep 09 10:49:31.177438 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.177293    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-model-cache\") pod \"63b7dec3-984a-419b-a976-f64998bbdfaf\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") "
Sep 09 10:49:31.177438 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.177319    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-tmp-dir\") pod \"63b7dec3-984a-419b-a976-f64998bbdfaf\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") "
Sep 09 10:49:31.177438 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.177352    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b7dec3-984a-419b-a976-f64998bbdfaf-tls-certs\") pod \"63b7dec3-984a-419b-a976-f64998bbdfaf\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") "
Sep 09 10:49:31.177438 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.177381    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-home\") pod \"63b7dec3-984a-419b-a976-f64998bbdfaf\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") "
Sep 09 10:49:31.177438 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.177414    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-dshm\") pod \"63b7dec3-984a-419b-a976-f64998bbdfaf\" (UID: \"63b7dec3-984a-419b-a976-f64998bbdfaf\") "
Sep 09 10:49:31.177702 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.177545    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "63b7dec3-984a-419b-a976-f64998bbdfaf" (UID: "63b7dec3-984a-419b-a976-f64998bbdfaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:31.177702 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.177552    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-model-cache" (OuterVolumeSpecName: "model-cache") pod "63b7dec3-984a-419b-a976-f64998bbdfaf" (UID: "63b7dec3-984a-419b-a976-f64998bbdfaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:31.177702 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.177678    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-home" (OuterVolumeSpecName: "home") pod "63b7dec3-984a-419b-a976-f64998bbdfaf" (UID: "63b7dec3-984a-419b-a976-f64998bbdfaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:31.177907 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.177886    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-model-cache\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:31.177964 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.177914    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-tmp-dir\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:31.177964 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.177927    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-home\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:31.179568 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.179547    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-dshm" (OuterVolumeSpecName: "dshm") pod "63b7dec3-984a-419b-a976-f64998bbdfaf" (UID: "63b7dec3-984a-419b-a976-f64998bbdfaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:31.179735 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.179644    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b7dec3-984a-419b-a976-f64998bbdfaf-kube-api-access-dj288" (OuterVolumeSpecName: "kube-api-access-dj288") pod "63b7dec3-984a-419b-a976-f64998bbdfaf" (UID: "63b7dec3-984a-419b-a976-f64998bbdfaf"). InnerVolumeSpecName "kube-api-access-dj288". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:49:31.179944 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.179914    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b7dec3-984a-419b-a976-f64998bbdfaf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63b7dec3-984a-419b-a976-f64998bbdfaf" (UID: "63b7dec3-984a-419b-a976-f64998bbdfaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:49:31.278708 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.278625    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj288\" (UniqueName: \"kubernetes.io/projected/63b7dec3-984a-419b-a976-f64998bbdfaf-kube-api-access-dj288\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:31.278708 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.278656    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b7dec3-984a-419b-a976-f64998bbdfaf-tls-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:31.278708 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.278665    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b7dec3-984a-419b-a976-f64998bbdfaf-dshm\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:31.944650 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.944607    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc" event={"ID":"63b7dec3-984a-419b-a976-f64998bbdfaf","Type":"ContainerDied","Data":"7a946f8f2a51375b5217497a9fe975d5d3d863cd9a0e156891f8372d1a0f47d5"}
Sep 09 10:49:31.945162 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.944682    2570 scope.go:117] "RemoveContainer" containerID="3f506dd7089f27264ad54fe1e17e62aed83c40f177e1b9b8b97d99739a83fa4b"
Sep 09 10:49:31.945162 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.944868    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"
Sep 09 10:49:31.972924 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.972889    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"]
Sep 09 10:49:31.977133 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:31.977111    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c02105/noten-model-c02105-kserve-79995cc547-cnkvc"]
Sep 09 10:49:32.839176 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:32.839149    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63b7dec3-984a-419b-a976-f64998bbdfaf" path="/var/lib/kubelet/pods/63b7dec3-984a-419b-a976-f64998bbdfaf/volumes"
Sep 09 10:49:35.911995 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:35.911940    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 10:49:35.912454 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:35.912046    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 10:49:37.372312 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.372276    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"]
Sep 09 10:49:37.372790 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.372536    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm" podUID="5b31e44a-a347-46b3-9399-bce12c382a67" containerName="maas-api" containerID="cri-o://21f8d8160bd773d89836012adc331bb05b389c7076d285473f9dae1b4a768a26" gracePeriod=30
Sep 09 10:49:37.405311 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.405284    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:37.502904 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.502871    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:37.602119 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.602092    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:37.618245 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.618225    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:49:37.641142 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.641075    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b31e44a-a347-46b3-9399-bce12c382a67-openshift-service-ca\") pod \"5b31e44a-a347-46b3-9399-bce12c382a67\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") "
Sep 09 10:49:37.641142 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.641119    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-maas-api-tls\") pod \"5b31e44a-a347-46b3-9399-bce12c382a67\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") "
Sep 09 10:49:37.641353 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.641203    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhc8g\" (UniqueName: \"kubernetes.io/projected/5b31e44a-a347-46b3-9399-bce12c382a67-kube-api-access-qhc8g\") pod \"5b31e44a-a347-46b3-9399-bce12c382a67\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") "
Sep 09 10:49:37.641410 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.641392    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-metrics-certs\") pod \"5b31e44a-a347-46b3-9399-bce12c382a67\" (UID: \"5b31e44a-a347-46b3-9399-bce12c382a67\") "
Sep 09 10:49:37.641828 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.641514    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b31e44a-a347-46b3-9399-bce12c382a67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b31e44a-a347-46b3-9399-bce12c382a67" (UID: "5b31e44a-a347-46b3-9399-bce12c382a67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:49:37.641828 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.641782    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b31e44a-a347-46b3-9399-bce12c382a67-openshift-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:37.646176 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.646145    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b31e44a-a347-46b3-9399-bce12c382a67-kube-api-access-qhc8g" (OuterVolumeSpecName: "kube-api-access-qhc8g") pod "5b31e44a-a347-46b3-9399-bce12c382a67" (UID: "5b31e44a-a347-46b3-9399-bce12c382a67"). InnerVolumeSpecName "kube-api-access-qhc8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:49:37.646350 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.646309    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b31e44a-a347-46b3-9399-bce12c382a67" (UID: "5b31e44a-a347-46b3-9399-bce12c382a67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:49:37.646585 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.646559    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b31e44a-a347-46b3-9399-bce12c382a67" (UID: "5b31e44a-a347-46b3-9399-bce12c382a67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:49:37.703013 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.702981    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:37.743373 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.743314    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:37.743373 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.743345    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b31e44a-a347-46b3-9399-bce12c382a67-maas-api-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:37.743373 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.743358    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhc8g\" (UniqueName: \"kubernetes.io/projected/5b31e44a-a347-46b3-9399-bce12c382a67-kube-api-access-qhc8g\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:49:37.983613 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.983514    2570 generic.go:356] "Generic (PLEG): container finished" podID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerID="95b966dc1777b7745de817054f8ce1f16881b2cfefdd281ede0062be3ef3b844" exitCode=0
Sep 09 10:49:37.983613 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.983593    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" event={"ID":"c391eaca-317f-4de8-994b-f6d246ae8f9b","Type":"ContainerDied","Data":"95b966dc1777b7745de817054f8ce1f16881b2cfefdd281ede0062be3ef3b844"}
Sep 09 10:49:37.984114 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.984088    2570 scope.go:117] "RemoveContainer" containerID="95b966dc1777b7745de817054f8ce1f16881b2cfefdd281ede0062be3ef3b844"
Sep 09 10:49:37.984833 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.984815    2570 generic.go:356] "Generic (PLEG): container finished" podID="5b31e44a-a347-46b3-9399-bce12c382a67" containerID="21f8d8160bd773d89836012adc331bb05b389c7076d285473f9dae1b4a768a26" exitCode=0
Sep 09 10:49:37.984916 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.984842    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm" event={"ID":"5b31e44a-a347-46b3-9399-bce12c382a67","Type":"ContainerDied","Data":"21f8d8160bd773d89836012adc331bb05b389c7076d285473f9dae1b4a768a26"}
Sep 09 10:49:37.984916 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.984865    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"
Sep 09 10:49:37.984916 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.984878    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm" event={"ID":"5b31e44a-a347-46b3-9399-bce12c382a67","Type":"ContainerDied","Data":"0b7edcedc3dc649a9820418f65635fbf5e103d47621739652535a7c9dc5aa0db"}
Sep 09 10:49:37.984916 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.984903    2570 scope.go:117] "RemoveContainer" containerID="21f8d8160bd773d89836012adc331bb05b389c7076d285473f9dae1b4a768a26"
Sep 09 10:49:37.985161 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.985135    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:49:37.985270 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.985251    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:49:37.985355 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.985305    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:49:37.997450 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.997432    2570 scope.go:117] "RemoveContainer" containerID="21f8d8160bd773d89836012adc331bb05b389c7076d285473f9dae1b4a768a26"
Sep 09 10:49:37.997730 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:49:37.997713    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f8d8160bd773d89836012adc331bb05b389c7076d285473f9dae1b4a768a26\": container with ID starting with 21f8d8160bd773d89836012adc331bb05b389c7076d285473f9dae1b4a768a26 not found: ID does not exist" containerID="21f8d8160bd773d89836012adc331bb05b389c7076d285473f9dae1b4a768a26"
Sep 09 10:49:37.997806 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:37.997737    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f8d8160bd773d89836012adc331bb05b389c7076d285473f9dae1b4a768a26"} err="failed to get container status \"21f8d8160bd773d89836012adc331bb05b389c7076d285473f9dae1b4a768a26\": rpc error: code = NotFound desc = could not find container \"21f8d8160bd773d89836012adc331bb05b389c7076d285473f9dae1b4a768a26\": container with ID starting with 21f8d8160bd773d89836012adc331bb05b389c7076d285473f9dae1b4a768a26 not found: ID does not exist"
Sep 09 10:49:38.035168 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:38.035100    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:38.041527 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:38.041505    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"]
Sep 09 10:49:38.060442 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:38.060417    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a090f42-7f6c5585bb-fjpjm"]
Sep 09 10:49:38.839965 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:38.839927    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b31e44a-a347-46b3-9399-bce12c382a67" path="/var/lib/kubelet/pods/5b31e44a-a347-46b3-9399-bce12c382a67/volumes"
Sep 09 10:49:38.992046 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:38.992009    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" event={"ID":"c391eaca-317f-4de8-994b-f6d246ae8f9b","Type":"ContainerStarted","Data":"bd2a08a8bb1bdb18381a2483ba141bc90063962ea3c5c9f6e2fd74a3b4c516c7"}
Sep 09 10:49:39.624780 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:39.624734    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:39.624975 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:39.624951    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 09 10:49:39.625036 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:39.625013    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 09 10:49:40.625824 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:40.625786    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 09 10:49:40.626196 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:40.625853    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 09 10:49:41.625477 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:41.625442    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 09 10:49:41.625628 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:41.625497    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 09 10:49:42.625016 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:42.624976    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 09 10:49:42.625473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:42.625044    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 09 10:49:43.251549 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:43.251495    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:43.625007 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:43.624926    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 09 10:49:43.625007 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:43.624991    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 09 10:49:44.438841 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:44.438806    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:44.513227 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:44.513176    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:44.629193 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:44.629159    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:44.629546 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:44.629511    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:44.630091 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:44.630074    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 10:49:44.721918 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:44.721837    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:44.969238 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:44.969206    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:45.466273 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:45.466238    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:45.746603 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:45.746572    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:45.914306 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:45.914276    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:51.645307 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:51.645273    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:52.012832 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:52.012756    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:52.494154 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:52.494120    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:52.594337 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:52.594305    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:53.500994 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:53.500922    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:56.911342 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:56.911309    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:56.968165 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:56.968134    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:58.605294 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:58.605262    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:58.706002 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:58.705974    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:59.591003 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:59.590970    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:49:59.925754 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:49:59.925670    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:00.667127 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:00.667095    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:02.849567 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:02.849531    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:02.935505 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:02.935449    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:03.984094 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:03.984065    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"]
Sep 09 10:50:03.984449 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:03.984437    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b31e44a-a347-46b3-9399-bce12c382a67" containerName="maas-api"
Sep 09 10:50:03.984502 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:03.984450    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b31e44a-a347-46b3-9399-bce12c382a67" containerName="maas-api"
Sep 09 10:50:03.984502 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:03.984467    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b7dec3-984a-419b-a976-f64998bbdfaf" containerName="main"
Sep 09 10:50:03.984502 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:03.984472    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b7dec3-984a-419b-a976-f64998bbdfaf" containerName="main"
Sep 09 10:50:03.984602 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:03.984531    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="63b7dec3-984a-419b-a976-f64998bbdfaf" containerName="main"
Sep 09 10:50:03.984602 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:03.984539    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b31e44a-a347-46b3-9399-bce12c382a67" containerName="maas-api"
Sep 09 10:50:03.988812 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:03.988791    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:03.991539 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:03.991511    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-62f893-serving-cert\""
Sep 09 10:50:04.003079 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.003055    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"]
Sep 09 10:50:04.099960 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.099926    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.100123 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.099965    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bjvm\" (UniqueName: \"kubernetes.io/projected/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-kube-api-access-6bjvm\") pod \"maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.100123 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.100062    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.100204 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.100137    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.200695 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.200663    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.200873 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.200703    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bjvm\" (UniqueName: \"kubernetes.io/projected/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-kube-api-access-6bjvm\") pod \"maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.200873 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.200829    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.200949 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.200928    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.201080 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:50:04.201064    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-serving-cert: secret "maas-api-e2e-shared-a-w2-62f893-serving-cert" not found
Sep 09 10:50:04.201156 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:50:04.201147    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-maas-api-tls podName:0d5434d4-fd82-4f46-b5a3-15aea93fecb3 nodeName:}" failed. No retries permitted until 2026-09-09 10:50:04.701132904 +0000 UTC m=+1444.364771861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-maas-api-tls") pod "maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265" (UID: "0d5434d4-fd82-4f46-b5a3-15aea93fecb3") : secret "maas-api-e2e-shared-a-w2-62f893-serving-cert" not found
Sep 09 10:50:04.201467 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.201449    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.203114 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.203097    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.210276 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.210252    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bjvm\" (UniqueName: \"kubernetes.io/projected/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-kube-api-access-6bjvm\") pod \"maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.369525 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.369481    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:04.674984 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.674897    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"]
Sep 09 10:50:04.678822 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.678801    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"
Sep 09 10:50:04.681957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.681932    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-62f893\""
Sep 09 10:50:04.682611 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.682587    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-62f893-dockercfg-mj6tg\""
Sep 09 10:50:04.688369 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.688345    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"]
Sep 09 10:50:04.706389 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.706361    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.708806 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.708782    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.807307 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.807269    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c8b2f1-b7f7-4a32-a450-0afb0b9345ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj\" (UID: \"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"
Sep 09 10:50:04.807608 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.807581    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwxn\" (UniqueName: \"kubernetes.io/projected/61c8b2f1-b7f7-4a32-a450-0afb0b9345ba-kube-api-access-2dwxn\") pod \"payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj\" (UID: \"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"
Sep 09 10:50:04.836548 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.836512    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:04.898942 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.898909    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:04.908564 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.908533    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c8b2f1-b7f7-4a32-a450-0afb0b9345ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj\" (UID: \"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"
Sep 09 10:50:04.908708 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.908598    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwxn\" (UniqueName: \"kubernetes.io/projected/61c8b2f1-b7f7-4a32-a450-0afb0b9345ba-kube-api-access-2dwxn\") pod \"payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj\" (UID: \"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"
Sep 09 10:50:04.909407 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.909388    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c8b2f1-b7f7-4a32-a450-0afb0b9345ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj\" (UID: \"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"
Sep 09 10:50:04.919163 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.919133    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwxn\" (UniqueName: \"kubernetes.io/projected/61c8b2f1-b7f7-4a32-a450-0afb0b9345ba-kube-api-access-2dwxn\") pod \"payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj\" (UID: \"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"
Sep 09 10:50:04.986917 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.986890    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:04.991246 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:04.991221    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"
Sep 09 10:50:05.172497 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:05.172465    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"]
Sep 09 10:50:05.174932 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:50:05.174906    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5434d4_fd82_4f46_b5a3_15aea93fecb3.slice/crio-3bc9ecbc932bd7b6f55ae552e12f4de9778981d85a9429a52947fe23fc4f8070 WatchSource:0}: Error finding container 3bc9ecbc932bd7b6f55ae552e12f4de9778981d85a9429a52947fe23fc4f8070: Status 404 returned error can't find the container with id 3bc9ecbc932bd7b6f55ae552e12f4de9778981d85a9429a52947fe23fc4f8070
Sep 09 10:50:05.272223 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:05.272072    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"]
Sep 09 10:50:05.274500 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:50:05.274474    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c8b2f1_b7f7_4a32_a450_0afb0b9345ba.slice/crio-996413ed185f23d1eeb8a746c455e670aca57dfbab46980248f8ec8f478a6a80 WatchSource:0}: Error finding container 996413ed185f23d1eeb8a746c455e670aca57dfbab46980248f8ec8f478a6a80: Status 404 returned error can't find the container with id 996413ed185f23d1eeb8a746c455e670aca57dfbab46980248f8ec8f478a6a80
Sep 09 10:50:06.131092 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:06.131061    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265" event={"ID":"0d5434d4-fd82-4f46-b5a3-15aea93fecb3","Type":"ContainerStarted","Data":"3bc9ecbc932bd7b6f55ae552e12f4de9778981d85a9429a52947fe23fc4f8070"}
Sep 09 10:50:06.132719 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:06.132697    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj" event={"ID":"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba","Type":"ContainerStarted","Data":"1a84f7b2b40b5def98b2b01b77ca83f42be7efd62b9bdb84cd1574294a3792d6"}
Sep 09 10:50:06.132896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:06.132743    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj" event={"ID":"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba","Type":"ContainerStarted","Data":"996413ed185f23d1eeb8a746c455e670aca57dfbab46980248f8ec8f478a6a80"}
Sep 09 10:50:06.132973 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:06.132893    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"
Sep 09 10:50:06.153007 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:06.152967    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj" podStartSLOduration=2.152953051 podStartE2EDuration="2.152953051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:50:06.151521383 +0000 UTC m=+1445.815160394" watchObservedRunningTime="2026-09-09 10:50:06.152953051 +0000 UTC m=+1445.816592071"
Sep 09 10:50:07.138610 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:07.138571    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265" event={"ID":"0d5434d4-fd82-4f46-b5a3-15aea93fecb3","Type":"ContainerStarted","Data":"1b361f3b5237214bc95ca9ac5f8664852e707c38026098b841203569153aabed"}
Sep 09 10:50:07.139012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:07.138812    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:07.162926 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:07.162880    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265" podStartSLOduration=3.239752049 podStartE2EDuration="4.162867536s" totalImagesPullingTime="923.115487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:50:07.160988195 +0000 UTC m=+1446.824627168" watchObservedRunningTime="2026-09-09 10:50:07.162867536 +0000 UTC m=+1446.826506509"
Sep 09 10:50:13.148670 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:13.148640    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:50:17.140931 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:17.140900    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"
Sep 09 10:50:27.693164 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:27.693126    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:28.031286 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:28.031254    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:28.355359 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:28.355263    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:28.769935 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:28.769908    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:31.902027 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:31.901993    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:31.996329 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:31.996301    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:33.209580 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:33.209547    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:33.314482 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:33.314448    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:33.543643 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:33.543611    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:33.766290 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:33.766259    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:33.935365 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:33.935285    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:34.927444 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:34.927407    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:41.337561 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:41.337523    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:46.177178 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:46.177142    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:50:56.898187 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:50:56.898152    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:51:01.087908 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:01.087876    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:51:01.091617 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:01.091596    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:51:01.093860 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:01.093844    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:51:01.097652 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:01.097638    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:51:10.030746 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.030714    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:51:10.328588 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.328518    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"]
Sep 09 10:51:10.332217 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.332202    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.334841 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.334822    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bbfd5f59-openshift-default-dockercfg-4bjjj\""
Sep 09 10:51:10.342708 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.342684    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"]
Sep 09 10:51:10.347370 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.347345    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:51:10.384163 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.384142    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7904c7b6-5fb8-42ea-81fb-3a8face75789-istiod-ca-cert\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.384254 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.384206    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-credential-socket\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.384296 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.384263    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-workload-certs\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.384332 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.384294    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-data\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.384375 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.384346    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk7tx\" (UniqueName: \"kubernetes.io/projected/7904c7b6-5fb8-42ea-81fb-3a8face75789-kube-api-access-mk7tx\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.384412 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.384376    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-envoy\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.384412 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.384400    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-podinfo\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.384476 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.384419    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-token\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.384512 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.384491    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-workload-socket\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.484991 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.484964    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk7tx\" (UniqueName: \"kubernetes.io/projected/7904c7b6-5fb8-42ea-81fb-3a8face75789-kube-api-access-mk7tx\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.485124 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.485003    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-envoy\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.485124 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.485031    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-podinfo\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.485124 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.485059    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-token\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.485124 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.485102    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-workload-socket\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.485359 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.485134    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7904c7b6-5fb8-42ea-81fb-3a8face75789-istiod-ca-cert\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.485359 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.485214    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-credential-socket\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.485359 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.485256    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-workload-certs\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.485359 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.485305    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-data\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.485574 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.485517    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-workload-socket\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.485798 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.485778    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-credential-socket\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.485934 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.485809    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-data\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.485934 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.485837    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-workload-certs\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.486098 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.486036    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7904c7b6-5fb8-42ea-81fb-3a8face75789-istiod-ca-cert\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.487316 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.487298    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-envoy\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.487750 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.487729    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-podinfo\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.496445 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.496419    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-token\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.496545 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.496505    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk7tx\" (UniqueName: \"kubernetes.io/projected/7904c7b6-5fb8-42ea-81fb-3a8face75789-kube-api-access-mk7tx\") pod \"e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") " pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.643402 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.643316    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:10.856592 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.856557    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:51:10.922111 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.922082    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"]
Sep 09 10:51:10.923218 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:51:10.923194    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7904c7b6_5fb8_42ea_81fb_3a8face75789.slice/crio-f61ff4a2c0129673fe0956b54adf9ce6df10918f31f3d2c2f64d21f4940ddfb5 WatchSource:0}: Error finding container f61ff4a2c0129673fe0956b54adf9ce6df10918f31f3d2c2f64d21f4940ddfb5: Status 404 returned error can't find the container with id f61ff4a2c0129673fe0956b54adf9ce6df10918f31f3d2c2f64d21f4940ddfb5
Sep 09 10:51:10.925357 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.925329    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:51:10.925434 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.925391    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:51:10.925434 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:10.925418    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:51:11.453825 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:11.453785    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" event={"ID":"7904c7b6-5fb8-42ea-81fb-3a8face75789","Type":"ContainerStarted","Data":"f3dc623b2ffb7482224fa134ae1db4ee3a1ea161e8375757f5296986d631cf70"}
Sep 09 10:51:11.454226 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:11.453857    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" event={"ID":"7904c7b6-5fb8-42ea-81fb-3a8face75789","Type":"ContainerStarted","Data":"f61ff4a2c0129673fe0956b54adf9ce6df10918f31f3d2c2f64d21f4940ddfb5"}
Sep 09 10:51:11.476370 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:11.476324    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" podStartSLOduration=1.476309892 podStartE2EDuration="1.476309892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:51:11.473640379 +0000 UTC m=+1511.137279355" watchObservedRunningTime="2026-09-09 10:51:11.476309892 +0000 UTC m=+1511.139948866"
Sep 09 10:51:11.644296 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:11.644270    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:11.649872 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:11.649847    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:12.458867 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:12.458838    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:12.459689 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:12.459668    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:13.381216 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:13.381175    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:51:13.466949 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:13.466921    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:51:14.508490 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.508459    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"]
Sep 09 10:51:14.513418 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.513390    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:14.516428 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.516411    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bbfd5f59-serving-cert\""
Sep 09 10:51:14.524614 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.524591    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"]
Sep 09 10:51:14.533722 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.533693    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:51:14.625857 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.625819    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqncv\" (UniqueName: \"kubernetes.io/projected/f105a388-35ea-4447-9668-b09960840975-kube-api-access-kqncv\") pod \"maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:14.626040 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.625882    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-maas-api-tls\") pod \"maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:14.626040 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.625908    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f105a388-35ea-4447-9668-b09960840975-openshift-service-ca\") pod \"maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:14.626040 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.625923    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-metrics-certs\") pod \"maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:14.634473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.634447    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:51:14.726985 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.726955    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqncv\" (UniqueName: \"kubernetes.io/projected/f105a388-35ea-4447-9668-b09960840975-kube-api-access-kqncv\") pod \"maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:14.727104 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.727005    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-maas-api-tls\") pod \"maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:14.727104 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.727029    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f105a388-35ea-4447-9668-b09960840975-openshift-service-ca\") pod \"maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:14.727104 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.727046    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-metrics-certs\") pod \"maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:14.727281 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:51:14.727160    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-serving-cert: secret "maas-api-e2e-mt-bbfd5f59-serving-cert" not found
Sep 09 10:51:14.727281 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:51:14.727227    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-maas-api-tls podName:f105a388-35ea-4447-9668-b09960840975 nodeName:}" failed. No retries permitted until 2026-09-09 10:51:15.227212278 +0000 UTC m=+1514.890851235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-maas-api-tls") pod "maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh" (UID: "f105a388-35ea-4447-9668-b09960840975") : secret "maas-api-e2e-mt-bbfd5f59-serving-cert" not found
Sep 09 10:51:14.727647 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.727631    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f105a388-35ea-4447-9668-b09960840975-openshift-service-ca\") pod \"maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:14.729537 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.729518    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-metrics-certs\") pod \"maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:14.736988 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:14.736969    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqncv\" (UniqueName: \"kubernetes.io/projected/f105a388-35ea-4447-9668-b09960840975-kube-api-access-kqncv\") pod \"maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:15.058502 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:15.058473    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:51:15.151806 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:15.151738    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:51:15.232938 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:15.232904    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-maas-api-tls\") pod \"maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:15.235861 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:15.235835    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-maas-api-tls\") pod \"maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:15.286867 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:15.286833    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:51:15.425562 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:15.425479    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:15.701340 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:15.701308    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"]
Sep 09 10:51:15.702899 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:51:15.702874    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf105a388_35ea_4447_9668_b09960840975.slice/crio-1a589cab416c89cbf7b5de8bbf75e71db588597b7d097f6d96b0183b25b243d0 WatchSource:0}: Error finding container 1a589cab416c89cbf7b5de8bbf75e71db588597b7d097f6d96b0183b25b243d0: Status 404 returned error can't find the container with id 1a589cab416c89cbf7b5de8bbf75e71db588597b7d097f6d96b0183b25b243d0
Sep 09 10:51:16.483832 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:16.483743    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh" event={"ID":"f105a388-35ea-4447-9668-b09960840975","Type":"ContainerStarted","Data":"924833ff382b0546c49e69c1257d20000d99967f9d143e2d0e5a3263639c10e3"}
Sep 09 10:51:16.483832 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:16.483818    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh" event={"ID":"f105a388-35ea-4447-9668-b09960840975","Type":"ContainerStarted","Data":"1a589cab416c89cbf7b5de8bbf75e71db588597b7d097f6d96b0183b25b243d0"}
Sep 09 10:51:16.484006 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:16.483902    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:16.502913 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:16.502869    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh" podStartSLOduration=2.087785405 podStartE2EDuration="2.502857085s" totalImagesPullingTime="415.07168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:51:16.500544004 +0000 UTC m=+1516.164182979" watchObservedRunningTime="2026-09-09 10:51:16.502857085 +0000 UTC m=+1516.166496059"
Sep 09 10:51:22.494470 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:22.494439    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:51:31.561384 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:31.561352    2570 generic.go:356] "Generic (PLEG): container finished" podID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerID="f3dc623b2ffb7482224fa134ae1db4ee3a1ea161e8375757f5296986d631cf70" exitCode=0
Sep 09 10:51:31.561788 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:31.561425    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" event={"ID":"7904c7b6-5fb8-42ea-81fb-3a8face75789","Type":"ContainerDied","Data":"f3dc623b2ffb7482224fa134ae1db4ee3a1ea161e8375757f5296986d631cf70"}
Sep 09 10:51:31.561868 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:31.561851    2570 scope.go:117] "RemoveContainer" containerID="f3dc623b2ffb7482224fa134ae1db4ee3a1ea161e8375757f5296986d631cf70"
Sep 09 10:51:31.562902 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:31.562873    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:51:31.562970 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:31.562940    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:51:31.563007 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:31.562988    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:51:32.570693 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:32.570660    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" event={"ID":"7904c7b6-5fb8-42ea-81fb-3a8face75789","Type":"ContainerStarted","Data":"e087b098645e11bb6f283a9e6afff9fd15d1e5020504bdbafd9a43f6514a53ea"}
Sep 09 10:51:32.644071 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:32.644035    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:32.644386 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:32.644358    2570 patch_prober.go:28] interesting pod/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 09 10:51:32.644471 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:32.644411    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 09 10:51:33.644921 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:33.644424    2570 patch_prober.go:28] interesting pod/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 09 10:51:33.644921 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:33.644488    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 09 10:51:34.644451 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:34.644414    2570 patch_prober.go:28] interesting pod/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 09 10:51:34.644653 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:34.644482    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 09 10:51:35.644045 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:35.644002    2570 patch_prober.go:28] interesting pod/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 09 10:51:35.644535 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:35.644079    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 09 10:51:36.644122 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:36.644091    2570 patch_prober.go:28] interesting pod/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 09 10:51:36.644674 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:36.644645    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 09 10:51:37.644406 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:37.644374    2570 patch_prober.go:28] interesting pod/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 09 10:51:37.644817 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:37.644437    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 09 10:51:38.362675 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:38.362635    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:51:38.648145 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:38.648071    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:38.648584 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:38.648342    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:38.648948 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:38.648933    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:51:42.199464 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:51:42.199429    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:52:01.001605 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:01.001576    2570 scope.go:117] "RemoveContainer" containerID="17729f2111c0c239c512c43abc7db845e812290ae02d19a87f61a7864c5e2751"
Sep 09 10:52:04.082533 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:04.082497    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:52:38.351218 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:38.351182    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:52:38.751448 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:38.751416    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:52:39.541813 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:39.541785    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:52:48.939272 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:48.939240    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:52:54.148967 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.148892    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"]
Sep 09 10:52:54.149405 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.149123    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh" podUID="f105a388-35ea-4447-9668-b09960840975" containerName="maas-api" containerID="cri-o://924833ff382b0546c49e69c1257d20000d99967f9d143e2d0e5a3263639c10e3" gracePeriod=30
Sep 09 10:52:54.307498 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.307460    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:52:54.406790 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.406690    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:52:54.411536 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.411514    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:52:54.482441 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.482404    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:52:54.507666 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.507642    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqncv\" (UniqueName: \"kubernetes.io/projected/f105a388-35ea-4447-9668-b09960840975-kube-api-access-kqncv\") pod \"f105a388-35ea-4447-9668-b09960840975\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") "
Sep 09 10:52:54.507796 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.507687    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f105a388-35ea-4447-9668-b09960840975-openshift-service-ca\") pod \"f105a388-35ea-4447-9668-b09960840975\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") "
Sep 09 10:52:54.507850 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.507813    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-metrics-certs\") pod \"f105a388-35ea-4447-9668-b09960840975\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") "
Sep 09 10:52:54.507892 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.507861    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-maas-api-tls\") pod \"f105a388-35ea-4447-9668-b09960840975\" (UID: \"f105a388-35ea-4447-9668-b09960840975\") "
Sep 09 10:52:54.508112 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.508086    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f105a388-35ea-4447-9668-b09960840975-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f105a388-35ea-4447-9668-b09960840975" (UID: "f105a388-35ea-4447-9668-b09960840975"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:52:54.508243 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.508229    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f105a388-35ea-4447-9668-b09960840975-openshift-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:52:54.509945 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.509920    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f105a388-35ea-4447-9668-b09960840975" (UID: "f105a388-35ea-4447-9668-b09960840975"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:52:54.509945 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.509941    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f105a388-35ea-4447-9668-b09960840975" (UID: "f105a388-35ea-4447-9668-b09960840975"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:52:54.510086 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.509954    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f105a388-35ea-4447-9668-b09960840975-kube-api-access-kqncv" (OuterVolumeSpecName: "kube-api-access-kqncv") pod "f105a388-35ea-4447-9668-b09960840975" (UID: "f105a388-35ea-4447-9668-b09960840975"). InnerVolumeSpecName "kube-api-access-kqncv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:52:54.608947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.608916    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:52:54.608947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.608939    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f105a388-35ea-4447-9668-b09960840975-maas-api-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:52:54.608947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.608951    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqncv\" (UniqueName: \"kubernetes.io/projected/f105a388-35ea-4447-9668-b09960840975-kube-api-access-kqncv\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:52:54.978405 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.978374    2570 generic.go:356] "Generic (PLEG): container finished" podID="f105a388-35ea-4447-9668-b09960840975" containerID="924833ff382b0546c49e69c1257d20000d99967f9d143e2d0e5a3263639c10e3" exitCode=0
Sep 09 10:52:54.978576 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.978423    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"
Sep 09 10:52:54.978576 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.978433    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh" event={"ID":"f105a388-35ea-4447-9668-b09960840975","Type":"ContainerDied","Data":"924833ff382b0546c49e69c1257d20000d99967f9d143e2d0e5a3263639c10e3"}
Sep 09 10:52:54.978576 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.978472    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh" event={"ID":"f105a388-35ea-4447-9668-b09960840975","Type":"ContainerDied","Data":"1a589cab416c89cbf7b5de8bbf75e71db588597b7d097f6d96b0183b25b243d0"}
Sep 09 10:52:54.978576 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.978493    2570 scope.go:117] "RemoveContainer" containerID="924833ff382b0546c49e69c1257d20000d99967f9d143e2d0e5a3263639c10e3"
Sep 09 10:52:54.987245 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.987221    2570 scope.go:117] "RemoveContainer" containerID="924833ff382b0546c49e69c1257d20000d99967f9d143e2d0e5a3263639c10e3"
Sep 09 10:52:54.987512 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:52:54.987488    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"924833ff382b0546c49e69c1257d20000d99967f9d143e2d0e5a3263639c10e3\": container with ID starting with 924833ff382b0546c49e69c1257d20000d99967f9d143e2d0e5a3263639c10e3 not found: ID does not exist" containerID="924833ff382b0546c49e69c1257d20000d99967f9d143e2d0e5a3263639c10e3"
Sep 09 10:52:54.987644 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:54.987517    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"924833ff382b0546c49e69c1257d20000d99967f9d143e2d0e5a3263639c10e3"} err="failed to get container status \"924833ff382b0546c49e69c1257d20000d99967f9d143e2d0e5a3263639c10e3\": rpc error: code = NotFound desc = could not find container \"924833ff382b0546c49e69c1257d20000d99967f9d143e2d0e5a3263639c10e3\": container with ID starting with 924833ff382b0546c49e69c1257d20000d99967f9d143e2d0e5a3263639c10e3 not found: ID does not exist"
Sep 09 10:52:55.000402 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:55.000368    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"]
Sep 09 10:52:55.006734 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:55.006707    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bbfd5f59-6cfc76bf49-cgcwh"]
Sep 09 10:52:56.840460 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:56.840428    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f105a388-35ea-4447-9668-b09960840975" path="/var/lib/kubelet/pods/f105a388-35ea-4447-9668-b09960840975/volumes"
Sep 09 10:52:57.736944 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:52:57.736905    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:02.812947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:02.812894    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:25.708861 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:25.708830    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:38.979950 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:38.979918    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:40.063497 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:40.063439    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:46.181637 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:46.181603    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:46.246131 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:46.246099    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:46.427451 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:46.427422    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"]
Sep 09 10:53:46.427696 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:46.427644    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerName="istio-proxy" containerID="cri-o://e087b098645e11bb6f283a9e6afff9fd15d1e5020504bdbafd9a43f6514a53ea" gracePeriod=30
Sep 09 10:53:46.448918 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:46.448856    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:47.721754 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:47.721721    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:47.837294 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:47.837260    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:48.023377 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:48.023351    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:48.588494 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:48.588463    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:48.686293 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:48.686261    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:48.786599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:48.786568    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:50.200814 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:50.200785    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:51.687417 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.687395    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:53:51.746103 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746082    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7904c7b6-5fb8-42ea-81fb-3a8face75789-istiod-ca-cert\") pod \"7904c7b6-5fb8-42ea-81fb-3a8face75789\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") "
Sep 09 10:53:51.746257 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746119    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-token\") pod \"7904c7b6-5fb8-42ea-81fb-3a8face75789\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") "
Sep 09 10:53:51.746257 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746174    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk7tx\" (UniqueName: \"kubernetes.io/projected/7904c7b6-5fb8-42ea-81fb-3a8face75789-kube-api-access-mk7tx\") pod \"7904c7b6-5fb8-42ea-81fb-3a8face75789\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") "
Sep 09 10:53:51.746257 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746240    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-envoy\") pod \"7904c7b6-5fb8-42ea-81fb-3a8face75789\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") "
Sep 09 10:53:51.746402 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746273    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-workload-certs\") pod \"7904c7b6-5fb8-42ea-81fb-3a8face75789\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") "
Sep 09 10:53:51.746402 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746314    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-data\") pod \"7904c7b6-5fb8-42ea-81fb-3a8face75789\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") "
Sep 09 10:53:51.746402 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746338    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-workload-socket\") pod \"7904c7b6-5fb8-42ea-81fb-3a8face75789\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") "
Sep 09 10:53:51.746558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746406    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-credential-socket\") pod \"7904c7b6-5fb8-42ea-81fb-3a8face75789\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") "
Sep 09 10:53:51.746558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746441    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-podinfo\") pod \"7904c7b6-5fb8-42ea-81fb-3a8face75789\" (UID: \"7904c7b6-5fb8-42ea-81fb-3a8face75789\") "
Sep 09 10:53:51.746558 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746501    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7904c7b6-5fb8-42ea-81fb-3a8face75789-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7904c7b6-5fb8-42ea-81fb-3a8face75789" (UID: "7904c7b6-5fb8-42ea-81fb-3a8face75789"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:53:51.746706 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746613    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-data" (OuterVolumeSpecName: "istio-data") pod "7904c7b6-5fb8-42ea-81fb-3a8face75789" (UID: "7904c7b6-5fb8-42ea-81fb-3a8face75789"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:53:51.746706 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746640    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7904c7b6-5fb8-42ea-81fb-3a8face75789" (UID: "7904c7b6-5fb8-42ea-81fb-3a8face75789"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:53:51.746836 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746811    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7904c7b6-5fb8-42ea-81fb-3a8face75789" (UID: "7904c7b6-5fb8-42ea-81fb-3a8face75789"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:53:51.746898 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746827    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7904c7b6-5fb8-42ea-81fb-3a8face75789" (UID: "7904c7b6-5fb8-42ea-81fb-3a8face75789"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:53:51.746898 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746879    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-workload-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:53:51.746898 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746897    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-data\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:53:51.747037 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.746911    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7904c7b6-5fb8-42ea-81fb-3a8face75789-istiod-ca-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:53:51.748633 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.748602    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7904c7b6-5fb8-42ea-81fb-3a8face75789" (UID: "7904c7b6-5fb8-42ea-81fb-3a8face75789"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:53:51.748741 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.748635    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7904c7b6-5fb8-42ea-81fb-3a8face75789-kube-api-access-mk7tx" (OuterVolumeSpecName: "kube-api-access-mk7tx") pod "7904c7b6-5fb8-42ea-81fb-3a8face75789" (UID: "7904c7b6-5fb8-42ea-81fb-3a8face75789"). InnerVolumeSpecName "kube-api-access-mk7tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:53:51.748867 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.748847    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-token" (OuterVolumeSpecName: "istio-token") pod "7904c7b6-5fb8-42ea-81fb-3a8face75789" (UID: "7904c7b6-5fb8-42ea-81fb-3a8face75789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:53:51.748915 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.748851    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7904c7b6-5fb8-42ea-81fb-3a8face75789" (UID: "7904c7b6-5fb8-42ea-81fb-3a8face75789"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:53:51.847701 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.847671    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-envoy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:53:51.847701 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.847696    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-workload-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:53:51.847701 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.847706    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7904c7b6-5fb8-42ea-81fb-3a8face75789-credential-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:53:51.847940 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.847715    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-podinfo\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:53:51.847940 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.847724    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7904c7b6-5fb8-42ea-81fb-3a8face75789-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:53:51.847940 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:51.847733    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk7tx\" (UniqueName: \"kubernetes.io/projected/7904c7b6-5fb8-42ea-81fb-3a8face75789-kube-api-access-mk7tx\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:53:52.260665 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:52.260626    2570 generic.go:356] "Generic (PLEG): container finished" podID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerID="e087b098645e11bb6f283a9e6afff9fd15d1e5020504bdbafd9a43f6514a53ea" exitCode=0
Sep 09 10:53:52.260831 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:52.260691    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"
Sep 09 10:53:52.260831 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:52.260707    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" event={"ID":"7904c7b6-5fb8-42ea-81fb-3a8face75789","Type":"ContainerDied","Data":"e087b098645e11bb6f283a9e6afff9fd15d1e5020504bdbafd9a43f6514a53ea"}
Sep 09 10:53:52.260831 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:52.260808    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx" event={"ID":"7904c7b6-5fb8-42ea-81fb-3a8face75789","Type":"ContainerDied","Data":"f61ff4a2c0129673fe0956b54adf9ce6df10918f31f3d2c2f64d21f4940ddfb5"}
Sep 09 10:53:52.260956 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:52.260833    2570 scope.go:117] "RemoveContainer" containerID="e087b098645e11bb6f283a9e6afff9fd15d1e5020504bdbafd9a43f6514a53ea"
Sep 09 10:53:52.270467 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:52.270452    2570 scope.go:117] "RemoveContainer" containerID="f3dc623b2ffb7482224fa134ae1db4ee3a1ea161e8375757f5296986d631cf70"
Sep 09 10:53:52.279527 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:52.279508    2570 scope.go:117] "RemoveContainer" containerID="e087b098645e11bb6f283a9e6afff9fd15d1e5020504bdbafd9a43f6514a53ea"
Sep 09 10:53:52.279805 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:53:52.279788    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e087b098645e11bb6f283a9e6afff9fd15d1e5020504bdbafd9a43f6514a53ea\": container with ID starting with e087b098645e11bb6f283a9e6afff9fd15d1e5020504bdbafd9a43f6514a53ea not found: ID does not exist" containerID="e087b098645e11bb6f283a9e6afff9fd15d1e5020504bdbafd9a43f6514a53ea"
Sep 09 10:53:52.279862 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:52.279811    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e087b098645e11bb6f283a9e6afff9fd15d1e5020504bdbafd9a43f6514a53ea"} err="failed to get container status \"e087b098645e11bb6f283a9e6afff9fd15d1e5020504bdbafd9a43f6514a53ea\": rpc error: code = NotFound desc = could not find container \"e087b098645e11bb6f283a9e6afff9fd15d1e5020504bdbafd9a43f6514a53ea\": container with ID starting with e087b098645e11bb6f283a9e6afff9fd15d1e5020504bdbafd9a43f6514a53ea not found: ID does not exist"
Sep 09 10:53:52.279862 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:52.279826    2570 scope.go:117] "RemoveContainer" containerID="f3dc623b2ffb7482224fa134ae1db4ee3a1ea161e8375757f5296986d631cf70"
Sep 09 10:53:52.280070 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:53:52.280053    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3dc623b2ffb7482224fa134ae1db4ee3a1ea161e8375757f5296986d631cf70\": container with ID starting with f3dc623b2ffb7482224fa134ae1db4ee3a1ea161e8375757f5296986d631cf70 not found: ID does not exist" containerID="f3dc623b2ffb7482224fa134ae1db4ee3a1ea161e8375757f5296986d631cf70"
Sep 09 10:53:52.280108 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:52.280077    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3dc623b2ffb7482224fa134ae1db4ee3a1ea161e8375757f5296986d631cf70"} err="failed to get container status \"f3dc623b2ffb7482224fa134ae1db4ee3a1ea161e8375757f5296986d631cf70\": rpc error: code = NotFound desc = could not find container \"f3dc623b2ffb7482224fa134ae1db4ee3a1ea161e8375757f5296986d631cf70\": container with ID starting with f3dc623b2ffb7482224fa134ae1db4ee3a1ea161e8375757f5296986d631cf70 not found: ID does not exist"
Sep 09 10:53:52.284341 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:52.284319    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"]
Sep 09 10:53:52.291213 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:52.291188    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bbfd5f59-openshift-default-66686bb8d8-b95rx"]
Sep 09 10:53:52.840268 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:52.840238    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" path="/var/lib/kubelet/pods/7904c7b6-5fb8-42ea-81fb-3a8face75789/volumes"
Sep 09 10:53:56.155126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:56.155091    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:56.237146 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:56.237117    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:56.877559 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:56.877533    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:53:58.826102 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:53:58.826074    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:03.454910 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:03.454876    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:03.561042 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:03.561008    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:03.662942 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:03.662910    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:13.584250 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:13.584215    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:13.648819 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:13.648784    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:14.145114 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:14.145079    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:14.363235 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:14.363202    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:14.863971 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:14.863938    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:29.537676 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:29.537549    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:30.704049 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:30.704000    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:35.375483 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:35.375456    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:41.699859 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:41.699822    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:41.771463 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:41.771436    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:42.112040 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.112006    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:42.381849 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.381776    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"]
Sep 09 10:54:42.382190 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.382177    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerName="istio-proxy"
Sep 09 10:54:42.382243 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.382191    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerName="istio-proxy"
Sep 09 10:54:42.382243 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.382211    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerName="istio-proxy"
Sep 09 10:54:42.382243 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.382216    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerName="istio-proxy"
Sep 09 10:54:42.382243 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.382227    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f105a388-35ea-4447-9668-b09960840975" containerName="maas-api"
Sep 09 10:54:42.382243 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.382232    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f105a388-35ea-4447-9668-b09960840975" containerName="maas-api"
Sep 09 10:54:42.382397 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.382293    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerName="istio-proxy"
Sep 09 10:54:42.382397 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.382301    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7904c7b6-5fb8-42ea-81fb-3a8face75789" containerName="istio-proxy"
Sep 09 10:54:42.382397 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.382307    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f105a388-35ea-4447-9668-b09960840975" containerName="maas-api"
Sep 09 10:54:42.384726 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.384711    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.387742 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.387711    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e5f9c5-openshift-default-dockercfg-qxpvk\""
Sep 09 10:54:42.398577 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.398553    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"]
Sep 09 10:54:42.411078 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.411053    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:42.505009 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.504969    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-podinfo\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.505230 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.505014    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-data\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.505230 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.505043    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-workload-socket\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.505230 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.505066    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-token\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.505230 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.505183    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ccaa45d-62c6-4046-95e6-80885aa208d5-istiod-ca-cert\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.505474 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.505242    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-envoy\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.505474 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.505272    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhb9g\" (UniqueName: \"kubernetes.io/projected/5ccaa45d-62c6-4046-95e6-80885aa208d5-kube-api-access-nhb9g\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.505474 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.505324    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-workload-certs\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.505474 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.505348    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-credential-socket\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.606526 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.606497    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-podinfo\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.606735 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.606536    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-data\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.606735 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.606577    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-workload-socket\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.606735 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.606601    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-token\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.606735 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.606635    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ccaa45d-62c6-4046-95e6-80885aa208d5-istiod-ca-cert\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.606735 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.606675    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-envoy\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.606735 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.606706    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhb9g\" (UniqueName: \"kubernetes.io/projected/5ccaa45d-62c6-4046-95e6-80885aa208d5-kube-api-access-nhb9g\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.607083 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.606786    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-workload-certs\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.607083 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.606819    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-credential-socket\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.607083 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.607026    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-workload-socket\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.607251 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.607084    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-data\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.607344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.607326    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-credential-socket\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.607422 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.607354    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-workload-certs\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.607422 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.607384    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ccaa45d-62c6-4046-95e6-80885aa208d5-istiod-ca-cert\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.609089 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.609067    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-podinfo\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.609210 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.609104    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-envoy\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.620305 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.620285    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-token\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.620676 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.620658    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhb9g\" (UniqueName: \"kubernetes.io/projected/5ccaa45d-62c6-4046-95e6-80885aa208d5-kube-api-access-nhb9g\") pod \"e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") " pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.694688 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.694606    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:42.937377 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.937322    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:42.969748 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.969726    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"]
Sep 09 10:54:42.971281 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:54:42.971258    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccaa45d_62c6_4046_95e6_80885aa208d5.slice/crio-bf827a311179db5bb306e7bef9586acc86b4134fc663fbe5a5eb6480b1c57b16 WatchSource:0}: Error finding container bf827a311179db5bb306e7bef9586acc86b4134fc663fbe5a5eb6480b1c57b16: Status 404 returned error can't find the container with id bf827a311179db5bb306e7bef9586acc86b4134fc663fbe5a5eb6480b1c57b16
Sep 09 10:54:42.973420 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.973391    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:54:42.973490 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.973454    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:54:42.973490 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:42.973483    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:54:43.528610 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:43.528573    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" event={"ID":"5ccaa45d-62c6-4046-95e6-80885aa208d5","Type":"ContainerStarted","Data":"6c47e58c902c504f55306d19366c650ebda9d0cf6418bf6375b6b91810cf2887"}
Sep 09 10:54:43.528833 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:43.528628    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" event={"ID":"5ccaa45d-62c6-4046-95e6-80885aa208d5","Type":"ContainerStarted","Data":"bf827a311179db5bb306e7bef9586acc86b4134fc663fbe5a5eb6480b1c57b16"}
Sep 09 10:54:43.553941 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:43.553899    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" podStartSLOduration=1.553883481 podStartE2EDuration="1.553883481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:54:43.552174014 +0000 UTC m=+1723.215812985" watchObservedRunningTime="2026-09-09 10:54:43.553883481 +0000 UTC m=+1723.217522454"
Sep 09 10:54:43.695362 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:43.695325    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:44.066005 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:44.065972    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:44.700049 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:44.700023    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:45.538739 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:45.538710    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:45.539641 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:45.539620    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:54:45.922623 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:45.922544    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:46.020956 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:46.020917    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:46.818412 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:46.818372    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"]
Sep 09 10:54:46.823389 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:46.823359    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:46.826003 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:46.825975    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e5f9c5-serving-cert\""
Sep 09 10:54:46.840103 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:46.840080    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"]
Sep 09 10:54:46.844927 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:46.844894    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:46.945089 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:46.945055    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:46.954282 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:46.954256    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2qhd\" (UniqueName: \"kubernetes.io/projected/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-kube-api-access-w2qhd\") pod \"maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:46.954349 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:46.954310    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-openshift-service-ca\") pod \"maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:46.954421 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:46.954350    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-maas-api-tls\") pod \"maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:46.954421 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:46.954367    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-metrics-certs\") pod \"maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:47.055563 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.055533    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qhd\" (UniqueName: \"kubernetes.io/projected/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-kube-api-access-w2qhd\") pod \"maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:47.055730 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.055586    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-openshift-service-ca\") pod \"maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:47.055730 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.055638    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-maas-api-tls\") pod \"maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:47.055730 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.055665    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-metrics-certs\") pod \"maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:47.056266 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.056234    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-openshift-service-ca\") pod \"maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:47.058113 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.058093    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-maas-api-tls\") pod \"maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:47.058253 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.058232    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-metrics-certs\") pod \"maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:47.064490 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.064466    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qhd\" (UniqueName: \"kubernetes.io/projected/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-kube-api-access-w2qhd\") pod \"maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:47.086718 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.086650    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:47.136725 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.136687    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:47.414586 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.414553    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"]
Sep 09 10:54:47.416438 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:54:47.416408    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6138ae9_1c05_4761_a1a9_5f24c2d49f49.slice/crio-7be32897b06cf7d7cbf09fb901b3baa3a86467806f6163e99ac8307a51780167 WatchSource:0}: Error finding container 7be32897b06cf7d7cbf09fb901b3baa3a86467806f6163e99ac8307a51780167: Status 404 returned error can't find the container with id 7be32897b06cf7d7cbf09fb901b3baa3a86467806f6163e99ac8307a51780167
Sep 09 10:54:47.418587 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.418567    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 10:54:47.475534 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.475499    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:47.549506 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.549471    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q" event={"ID":"d6138ae9-1c05-4761-a1a9-5f24c2d49f49","Type":"ContainerStarted","Data":"7be32897b06cf7d7cbf09fb901b3baa3a86467806f6163e99ac8307a51780167"}
Sep 09 10:54:47.590810 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:47.590761    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:54:48.555664 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:48.555628    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q" event={"ID":"d6138ae9-1c05-4761-a1a9-5f24c2d49f49","Type":"ContainerStarted","Data":"cf86c0ed765e5ea0d66cc8eb70a45e28862b02283883dea6e7a08a67ccd2973d"}
Sep 09 10:54:48.556065 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:48.555737    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:54:48.576717 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:48.576670    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q" podStartSLOduration=1.929813111 podStartE2EDuration="2.57665505s" totalImagesPullingTime="646.841939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:54:48.575081827 +0000 UTC m=+1728.238720801" watchObservedRunningTime="2026-09-09 10:54:48.57665505 +0000 UTC m=+1728.240294024"
Sep 09 10:54:54.568547 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:54:54.568509    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:55:01.579952 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:01.579914    2570 patch_prober.go:28] interesting pod/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 10:55:01.580421 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:01.580004    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 10:55:03.631730 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:03.631694    2570 generic.go:356] "Generic (PLEG): container finished" podID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerID="6c47e58c902c504f55306d19366c650ebda9d0cf6418bf6375b6b91810cf2887" exitCode=0
Sep 09 10:55:03.632281 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:03.631782    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" event={"ID":"5ccaa45d-62c6-4046-95e6-80885aa208d5","Type":"ContainerDied","Data":"6c47e58c902c504f55306d19366c650ebda9d0cf6418bf6375b6b91810cf2887"}
Sep 09 10:55:03.632354 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:03.632314    2570 scope.go:117] "RemoveContainer" containerID="6c47e58c902c504f55306d19366c650ebda9d0cf6418bf6375b6b91810cf2887"
Sep 09 10:55:03.633484 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:03.633458    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:55:03.633553 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:03.633528    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:55:03.633591 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:03.633558    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:55:04.639232 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:04.639200    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" event={"ID":"5ccaa45d-62c6-4046-95e6-80885aa208d5","Type":"ContainerStarted","Data":"ecccd1377612dc8673ed5a04a5b472f58eab963253e7b3a32c302cbdacff58af"}
Sep 09 10:55:04.695370 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:04.695335    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:55:04.695566 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:04.695533    2570 patch_prober.go:28] interesting pod/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 09 10:55:04.695643 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:04.695585    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 09 10:55:05.695707 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:05.695675    2570 patch_prober.go:28] interesting pod/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 09 10:55:05.696103 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:05.695731    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 09 10:55:06.695239 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:06.695206    2570 patch_prober.go:28] interesting pod/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 09 10:55:06.695435 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:06.695265    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 09 10:55:07.695784 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:07.695740    2570 patch_prober.go:28] interesting pod/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 09 10:55:07.696236 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:07.695814    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 09 10:55:08.694943 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:08.694907    2570 patch_prober.go:28] interesting pod/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 09 10:55:08.695125 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:08.694960    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 09 10:55:09.695451 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:09.695410    2570 patch_prober.go:28] interesting pod/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 09 10:55:09.695929 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:09.695470    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 09 10:55:10.495898 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:10.495867    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:10.698501 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:10.698473    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:55:10.698956 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:10.698936    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:55:10.699466 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:10.699447    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:55:10.976988 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:10.976905    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:11.292712 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:11.292679    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:11.392884 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:11.392852    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:14.757804 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:14.757742    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:14.856052 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:14.856027    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:15.938790 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:15.938735    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:16.039176 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.039147    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:16.248116 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.248083    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:16.450988 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.450930    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:16.548079 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.547996    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:16.636290 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.636256    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"]
Sep 09 10:55:16.638694 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.638677    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"
Sep 09 10:55:16.642915 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.642898    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-798524\""
Sep 09 10:55:16.643007 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.642984    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-798524-dockercfg-bp4f8\""
Sep 09 10:55:16.652223 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.652201    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"]
Sep 09 10:55:16.753270 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.753236    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77e5a58-b702-46ec-a2ae-e8a8603e5a1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2\" (UID: \"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"
Sep 09 10:55:16.753438 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.753285    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd648\" (UniqueName: \"kubernetes.io/projected/e77e5a58-b702-46ec-a2ae-e8a8603e5a1d-kube-api-access-bd648\") pod \"payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2\" (UID: \"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"
Sep 09 10:55:16.854331 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.854250    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77e5a58-b702-46ec-a2ae-e8a8603e5a1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2\" (UID: \"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"
Sep 09 10:55:16.854331 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.854290    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd648\" (UniqueName: \"kubernetes.io/projected/e77e5a58-b702-46ec-a2ae-e8a8603e5a1d-kube-api-access-bd648\") pod \"payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2\" (UID: \"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"
Sep 09 10:55:16.854917 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.854898    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77e5a58-b702-46ec-a2ae-e8a8603e5a1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2\" (UID: \"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"
Sep 09 10:55:16.864480 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.864459    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd648\" (UniqueName: \"kubernetes.io/projected/e77e5a58-b702-46ec-a2ae-e8a8603e5a1d-kube-api-access-bd648\") pod \"payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2\" (UID: \"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"
Sep 09 10:55:16.948575 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:16.948545    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"
Sep 09 10:55:17.222599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:17.222569    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"]
Sep 09 10:55:17.224237 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:55:17.224212    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77e5a58_b702_46ec_a2ae_e8a8603e5a1d.slice/crio-182a7f798b4c7e398d9d1e43b3a56e6188bae5b4743e7836b3f2119a99dcf632 WatchSource:0}: Error finding container 182a7f798b4c7e398d9d1e43b3a56e6188bae5b4743e7836b3f2119a99dcf632: Status 404 returned error can't find the container with id 182a7f798b4c7e398d9d1e43b3a56e6188bae5b4743e7836b3f2119a99dcf632
Sep 09 10:55:17.719067 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:17.719034    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2" event={"ID":"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d","Type":"ContainerStarted","Data":"f80349a10e09c5dfd8cde2be2105ab8524adba3a0af3e495115459c07a29d689"}
Sep 09 10:55:17.719246 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:17.719084    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2" event={"ID":"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d","Type":"ContainerStarted","Data":"182a7f798b4c7e398d9d1e43b3a56e6188bae5b4743e7836b3f2119a99dcf632"}
Sep 09 10:55:17.719246 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:17.719228    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"
Sep 09 10:55:17.739590 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:17.739546    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2" podStartSLOduration=1.7395334500000001 podStartE2EDuration="1.73953345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:55:17.737150043 +0000 UTC m=+1757.400789017" watchObservedRunningTime="2026-09-09 10:55:17.73953345 +0000 UTC m=+1757.403172423"
Sep 09 10:55:20.013136 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:20.013096    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:20.116065 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:20.116027    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:25.473999 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:25.473960    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:28.725655 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:28.725628    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"
Sep 09 10:55:37.778628 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:37.778593    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:39.820787 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:39.820737    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:42.957911 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:42.957869    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:44.902203 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.902173    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"]
Sep 09 10:55:44.906347 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.906326    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:44.910731 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.910703    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-535b75\"/\"default-dockercfg-4nnfx\""
Sep 09 10:55:44.910731 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.910725    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-535b75\"/\"kube-root-ca.crt\""
Sep 09 10:55:44.911357 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.910703    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-535b75\"/\"openshift-service-ca.crt\""
Sep 09 10:55:44.911357 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.910703    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-535b75\"/\"rate-test-model-535b75-kserve-self-signed-certs\""
Sep 09 10:55:44.915884 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.915864    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"]
Sep 09 10:55:44.925561 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.925539    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l986s\" (UniqueName: \"kubernetes.io/projected/8701c907-c853-4ce7-981d-37c36aa9c4d0-kube-api-access-l986s\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:44.925663 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.925579    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-tmp-dir\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:44.925709 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.925693    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-dshm\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:44.925751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.925722    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-home\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:44.925751 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.925744    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8701c907-c853-4ce7-981d-37c36aa9c4d0-tls-certs\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:44.925855 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.925812    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-model-cache\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:44.935995 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:44.935973    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:45.026599 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.026574    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-dshm\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:45.026727 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.026608    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-home\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:45.026727 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.026627    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8701c907-c853-4ce7-981d-37c36aa9c4d0-tls-certs\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:45.026727 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.026655    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-model-cache\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:45.026727 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.026700    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l986s\" (UniqueName: \"kubernetes.io/projected/8701c907-c853-4ce7-981d-37c36aa9c4d0-kube-api-access-l986s\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:45.026727 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.026722    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-tmp-dir\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:45.027070 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.027052    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-model-cache\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:45.027151 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.027082    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-home\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:45.027151 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.027139    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-tmp-dir\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:45.028746 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.028727    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-dshm\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:45.029012 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.028994    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8701c907-c853-4ce7-981d-37c36aa9c4d0-tls-certs\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:45.035645 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.035622    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l986s\" (UniqueName: \"kubernetes.io/projected/8701c907-c853-4ce7-981d-37c36aa9c4d0-kube-api-access-l986s\") pod \"rate-test-model-535b75-kserve-5b4697c87f-rlcb9\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") " pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:45.216434 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.216350    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:45.498026 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.498002    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"]
Sep 09 10:55:45.500222 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:55:45.500192    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8701c907_c853_4ce7_981d_37c36aa9c4d0.slice/crio-3602f0107489298dbad5e28573b2ac8a4bc2ed98a7552f8c4130e62715db7344 WatchSource:0}: Error finding container 3602f0107489298dbad5e28573b2ac8a4bc2ed98a7552f8c4130e62715db7344: Status 404 returned error can't find the container with id 3602f0107489298dbad5e28573b2ac8a4bc2ed98a7552f8c4130e62715db7344
Sep 09 10:55:45.865432 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.865348    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9" event={"ID":"8701c907-c853-4ce7-981d-37c36aa9c4d0","Type":"ContainerStarted","Data":"016daa171861841d020db3bbfb6ff4b080ebb0c09c6e675f001c8a1a8934d766"}
Sep 09 10:55:45.865432 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.865391    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9" event={"ID":"8701c907-c853-4ce7-981d-37c36aa9c4d0","Type":"ContainerStarted","Data":"3602f0107489298dbad5e28573b2ac8a4bc2ed98a7552f8c4130e62715db7344"}
Sep 09 10:55:45.889952 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:45.889905    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9" podStartSLOduration=1.8898911840000001 podStartE2EDuration="1.889891184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:55:45.888688076 +0000 UTC m=+1785.552327050" watchObservedRunningTime="2026-09-09 10:55:45.889891184 +0000 UTC m=+1785.553530197"
Sep 09 10:55:49.871967 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:49.871933    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:49.972231 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:49.972202    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:54.902386 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:54.902287    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:55.216722 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:55.216640    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:55.216722 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:55.216688    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:55.229548 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:55.229525    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:55.929007 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:55.928974    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:55:58.344865 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.344827    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"]
Sep 09 10:55:58.345349 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.345078    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q" podUID="d6138ae9-1c05-4761-a1a9-5f24c2d49f49" containerName="maas-api" containerID="cri-o://cf86c0ed765e5ea0d66cc8eb70a45e28862b02283883dea6e7a08a67ccd2973d" gracePeriod=30
Sep 09 10:55:58.403470 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.403434    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:58.483145 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.483105    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:58.553495 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.553463    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:58.591533 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.591511    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:55:58.632877 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.632781    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:58.656519 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.656493    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2qhd\" (UniqueName: \"kubernetes.io/projected/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-kube-api-access-w2qhd\") pod \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") "
Sep 09 10:55:58.656644 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.656524    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-maas-api-tls\") pod \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") "
Sep 09 10:55:58.656644 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.656573    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-metrics-certs\") pod \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") "
Sep 09 10:55:58.656732 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.656710    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-openshift-service-ca\") pod \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\" (UID: \"d6138ae9-1c05-4761-a1a9-5f24c2d49f49\") "
Sep 09 10:55:58.657091 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.657060    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6138ae9-1c05-4761-a1a9-5f24c2d49f49" (UID: "d6138ae9-1c05-4761-a1a9-5f24c2d49f49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:55:58.657230 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.657155    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-openshift-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:55:58.658618 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.658593    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6138ae9-1c05-4761-a1a9-5f24c2d49f49" (UID: "d6138ae9-1c05-4761-a1a9-5f24c2d49f49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:55:58.658696 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.658649    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d6138ae9-1c05-4761-a1a9-5f24c2d49f49" (UID: "d6138ae9-1c05-4761-a1a9-5f24c2d49f49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:55:58.658696 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.658666    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-kube-api-access-w2qhd" (OuterVolumeSpecName: "kube-api-access-w2qhd") pod "d6138ae9-1c05-4761-a1a9-5f24c2d49f49" (UID: "d6138ae9-1c05-4761-a1a9-5f24c2d49f49"). InnerVolumeSpecName "kube-api-access-w2qhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:55:58.700726 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.700702    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:55:58.758823 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.758759    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2qhd\" (UniqueName: \"kubernetes.io/projected/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-kube-api-access-w2qhd\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:55:58.758823 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.758825    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-maas-api-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:55:58.759070 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.758838    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6138ae9-1c05-4761-a1a9-5f24c2d49f49-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:55:58.933202 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.933128    2570 generic.go:356] "Generic (PLEG): container finished" podID="d6138ae9-1c05-4761-a1a9-5f24c2d49f49" containerID="cf86c0ed765e5ea0d66cc8eb70a45e28862b02283883dea6e7a08a67ccd2973d" exitCode=0
Sep 09 10:55:58.933202 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.933168    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"
Sep 09 10:55:58.933414 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.933214    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q" event={"ID":"d6138ae9-1c05-4761-a1a9-5f24c2d49f49","Type":"ContainerDied","Data":"cf86c0ed765e5ea0d66cc8eb70a45e28862b02283883dea6e7a08a67ccd2973d"}
Sep 09 10:55:58.933414 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.933269    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q" event={"ID":"d6138ae9-1c05-4761-a1a9-5f24c2d49f49","Type":"ContainerDied","Data":"7be32897b06cf7d7cbf09fb901b3baa3a86467806f6163e99ac8307a51780167"}
Sep 09 10:55:58.933414 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.933302    2570 scope.go:117] "RemoveContainer" containerID="cf86c0ed765e5ea0d66cc8eb70a45e28862b02283883dea6e7a08a67ccd2973d"
Sep 09 10:55:58.942136 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.942119    2570 scope.go:117] "RemoveContainer" containerID="cf86c0ed765e5ea0d66cc8eb70a45e28862b02283883dea6e7a08a67ccd2973d"
Sep 09 10:55:58.942382 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:55:58.942358    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf86c0ed765e5ea0d66cc8eb70a45e28862b02283883dea6e7a08a67ccd2973d\": container with ID starting with cf86c0ed765e5ea0d66cc8eb70a45e28862b02283883dea6e7a08a67ccd2973d not found: ID does not exist" containerID="cf86c0ed765e5ea0d66cc8eb70a45e28862b02283883dea6e7a08a67ccd2973d"
Sep 09 10:55:58.942455 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.942389    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf86c0ed765e5ea0d66cc8eb70a45e28862b02283883dea6e7a08a67ccd2973d"} err="failed to get container status \"cf86c0ed765e5ea0d66cc8eb70a45e28862b02283883dea6e7a08a67ccd2973d\": rpc error: code = NotFound desc = could not find container \"cf86c0ed765e5ea0d66cc8eb70a45e28862b02283883dea6e7a08a67ccd2973d\": container with ID starting with cf86c0ed765e5ea0d66cc8eb70a45e28862b02283883dea6e7a08a67ccd2973d not found: ID does not exist"
Sep 09 10:55:58.951490 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.951469    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"]
Sep 09 10:55:58.956936 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.956904    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e5f9c5-58c98fd858-hlg8q"]
Sep 09 10:55:58.998453 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:55:58.998426    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:00.841219 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:00.841179    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6138ae9-1c05-4761-a1a9-5f24c2d49f49" path="/var/lib/kubelet/pods/d6138ae9-1c05-4761-a1a9-5f24c2d49f49/volumes"
Sep 09 10:56:01.120245 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:01.120166    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:56:01.124114 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:01.124090    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 10:56:01.126750 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:01.126731    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:56:01.130331 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:01.130316    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 10:56:05.459581 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:05.459553    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"]
Sep 09 10:56:05.460023 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:05.459780    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy" containerID="cri-o://ecccd1377612dc8673ed5a04a5b472f58eab963253e7b3a32c302cbdacff58af" gracePeriod=30
Sep 09 10:56:06.597144 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:06.597106    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:06.698579 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:06.698547    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:10.713042 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.713017    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:56:10.871986 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.871958    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-envoy\") pod \"5ccaa45d-62c6-4046-95e6-80885aa208d5\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") "
Sep 09 10:56:10.872162 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872001    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhb9g\" (UniqueName: \"kubernetes.io/projected/5ccaa45d-62c6-4046-95e6-80885aa208d5-kube-api-access-nhb9g\") pod \"5ccaa45d-62c6-4046-95e6-80885aa208d5\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") "
Sep 09 10:56:10.872162 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872059    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-workload-socket\") pod \"5ccaa45d-62c6-4046-95e6-80885aa208d5\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") "
Sep 09 10:56:10.872162 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872105    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-workload-certs\") pod \"5ccaa45d-62c6-4046-95e6-80885aa208d5\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") "
Sep 09 10:56:10.872330 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872164    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-credential-socket\") pod \"5ccaa45d-62c6-4046-95e6-80885aa208d5\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") "
Sep 09 10:56:10.872330 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872208    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-podinfo\") pod \"5ccaa45d-62c6-4046-95e6-80885aa208d5\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") "
Sep 09 10:56:10.872330 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872255    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-data\") pod \"5ccaa45d-62c6-4046-95e6-80885aa208d5\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") "
Sep 09 10:56:10.872330 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872285    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-token\") pod \"5ccaa45d-62c6-4046-95e6-80885aa208d5\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") "
Sep 09 10:56:10.872330 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872296    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ccaa45d-62c6-4046-95e6-80885aa208d5" (UID: "5ccaa45d-62c6-4046-95e6-80885aa208d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:10.872638 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872349    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ccaa45d-62c6-4046-95e6-80885aa208d5-istiod-ca-cert\") pod \"5ccaa45d-62c6-4046-95e6-80885aa208d5\" (UID: \"5ccaa45d-62c6-4046-95e6-80885aa208d5\") "
Sep 09 10:56:10.872638 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872394    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ccaa45d-62c6-4046-95e6-80885aa208d5" (UID: "5ccaa45d-62c6-4046-95e6-80885aa208d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:10.872638 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872415    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ccaa45d-62c6-4046-95e6-80885aa208d5" (UID: "5ccaa45d-62c6-4046-95e6-80885aa208d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:10.872826 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872723    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-workload-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:10.872826 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872745    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-workload-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:10.872826 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872759    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-credential-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:10.872826 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872780    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ccaa45d-62c6-4046-95e6-80885aa208d5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ccaa45d-62c6-4046-95e6-80885aa208d5" (UID: "5ccaa45d-62c6-4046-95e6-80885aa208d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:56:10.873005 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.872953    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ccaa45d-62c6-4046-95e6-80885aa208d5" (UID: "5ccaa45d-62c6-4046-95e6-80885aa208d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:10.874324 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.874303    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ccaa45d-62c6-4046-95e6-80885aa208d5" (UID: "5ccaa45d-62c6-4046-95e6-80885aa208d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:56:10.874409 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.874363    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccaa45d-62c6-4046-95e6-80885aa208d5-kube-api-access-nhb9g" (OuterVolumeSpecName: "kube-api-access-nhb9g") pod "5ccaa45d-62c6-4046-95e6-80885aa208d5" (UID: "5ccaa45d-62c6-4046-95e6-80885aa208d5"). InnerVolumeSpecName "kube-api-access-nhb9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:56:10.874515 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.874499    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ccaa45d-62c6-4046-95e6-80885aa208d5" (UID: "5ccaa45d-62c6-4046-95e6-80885aa208d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:10.874717 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.874689    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ccaa45d-62c6-4046-95e6-80885aa208d5" (UID: "5ccaa45d-62c6-4046-95e6-80885aa208d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:56:10.973616 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.973595    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhb9g\" (UniqueName: \"kubernetes.io/projected/5ccaa45d-62c6-4046-95e6-80885aa208d5-kube-api-access-nhb9g\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:10.973616 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.973615    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-podinfo\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:10.973752 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.973625    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-data\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:10.973752 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.973633    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:10.973752 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.973640    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ccaa45d-62c6-4046-95e6-80885aa208d5-istiod-ca-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:10.973752 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:10.973647    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ccaa45d-62c6-4046-95e6-80885aa208d5-istio-envoy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:11.002217 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.002192    2570 generic.go:356] "Generic (PLEG): container finished" podID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerID="ecccd1377612dc8673ed5a04a5b472f58eab963253e7b3a32c302cbdacff58af" exitCode=0
Sep 09 10:56:11.002305 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.002235    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"
Sep 09 10:56:11.002305 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.002282    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" event={"ID":"5ccaa45d-62c6-4046-95e6-80885aa208d5","Type":"ContainerDied","Data":"ecccd1377612dc8673ed5a04a5b472f58eab963253e7b3a32c302cbdacff58af"}
Sep 09 10:56:11.002390 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.002335    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" event={"ID":"5ccaa45d-62c6-4046-95e6-80885aa208d5","Type":"ContainerDied","Data":"bf827a311179db5bb306e7bef9586acc86b4134fc663fbe5a5eb6480b1c57b16"}
Sep 09 10:56:11.002390 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.002356    2570 scope.go:117] "RemoveContainer" containerID="ecccd1377612dc8673ed5a04a5b472f58eab963253e7b3a32c302cbdacff58af"
Sep 09 10:56:11.011238 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.011223    2570 scope.go:117] "RemoveContainer" containerID="6c47e58c902c504f55306d19366c650ebda9d0cf6418bf6375b6b91810cf2887"
Sep 09 10:56:11.018845 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.018830    2570 scope.go:117] "RemoveContainer" containerID="ecccd1377612dc8673ed5a04a5b472f58eab963253e7b3a32c302cbdacff58af"
Sep 09 10:56:11.019099 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:56:11.019080    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecccd1377612dc8673ed5a04a5b472f58eab963253e7b3a32c302cbdacff58af\": container with ID starting with ecccd1377612dc8673ed5a04a5b472f58eab963253e7b3a32c302cbdacff58af not found: ID does not exist" containerID="ecccd1377612dc8673ed5a04a5b472f58eab963253e7b3a32c302cbdacff58af"
Sep 09 10:56:11.019148 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.019105    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecccd1377612dc8673ed5a04a5b472f58eab963253e7b3a32c302cbdacff58af"} err="failed to get container status \"ecccd1377612dc8673ed5a04a5b472f58eab963253e7b3a32c302cbdacff58af\": rpc error: code = NotFound desc = could not find container \"ecccd1377612dc8673ed5a04a5b472f58eab963253e7b3a32c302cbdacff58af\": container with ID starting with ecccd1377612dc8673ed5a04a5b472f58eab963253e7b3a32c302cbdacff58af not found: ID does not exist"
Sep 09 10:56:11.019148 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.019120    2570 scope.go:117] "RemoveContainer" containerID="6c47e58c902c504f55306d19366c650ebda9d0cf6418bf6375b6b91810cf2887"
Sep 09 10:56:11.019323 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:56:11.019308    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c47e58c902c504f55306d19366c650ebda9d0cf6418bf6375b6b91810cf2887\": container with ID starting with 6c47e58c902c504f55306d19366c650ebda9d0cf6418bf6375b6b91810cf2887 not found: ID does not exist" containerID="6c47e58c902c504f55306d19366c650ebda9d0cf6418bf6375b6b91810cf2887"
Sep 09 10:56:11.019367 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.019327    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c47e58c902c504f55306d19366c650ebda9d0cf6418bf6375b6b91810cf2887"} err="failed to get container status \"6c47e58c902c504f55306d19366c650ebda9d0cf6418bf6375b6b91810cf2887\": rpc error: code = NotFound desc = could not find container \"6c47e58c902c504f55306d19366c650ebda9d0cf6418bf6375b6b91810cf2887\": container with ID starting with 6c47e58c902c504f55306d19366c650ebda9d0cf6418bf6375b6b91810cf2887 not found: ID does not exist"
Sep 09 10:56:11.033752 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.033724    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"]
Sep 09 10:56:11.036246 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.036222    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7"]
Sep 09 10:56:11.700149 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.700115    2570 patch_prober.go:28] interesting pod/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 10:56:11.700325 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:11.700182    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e5f9c5-openshift-default-6cc499c474-n27n7" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 10:56:12.839700 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:12.839665    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" path="/var/lib/kubelet/pods/5ccaa45d-62c6-4046-95e6-80885aa208d5/volumes"
Sep 09 10:56:21.426090 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:21.426055    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:21.459194 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:21.459163    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"]
Sep 09 10:56:21.459434 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:21.459400    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2" podUID="e77e5a58-b702-46ec-a2ae-e8a8603e5a1d" containerName="payload-pre-processing" containerID="cri-o://f80349a10e09c5dfd8cde2be2105ab8524adba3a0af3e495115459c07a29d689" gracePeriod=30
Sep 09 10:56:21.497713 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:21.497674    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:21.628611 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:21.628439    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:21.707955 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:21.707931    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"
Sep 09 10:56:21.723362 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:21.723339    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:21.880953 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:21.880916    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd648\" (UniqueName: \"kubernetes.io/projected/e77e5a58-b702-46ec-a2ae-e8a8603e5a1d-kube-api-access-bd648\") pod \"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d\" (UID: \"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d\") "
Sep 09 10:56:21.881142 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:21.880981    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77e5a58-b702-46ec-a2ae-e8a8603e5a1d-plugins-config-volume\") pod \"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d\" (UID: \"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d\") "
Sep 09 10:56:21.881388 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:21.881366    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77e5a58-b702-46ec-a2ae-e8a8603e5a1d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e77e5a58-b702-46ec-a2ae-e8a8603e5a1d" (UID: "e77e5a58-b702-46ec-a2ae-e8a8603e5a1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:56:21.883134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:21.883108    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77e5a58-b702-46ec-a2ae-e8a8603e5a1d-kube-api-access-bd648" (OuterVolumeSpecName: "kube-api-access-bd648") pod "e77e5a58-b702-46ec-a2ae-e8a8603e5a1d" (UID: "e77e5a58-b702-46ec-a2ae-e8a8603e5a1d"). InnerVolumeSpecName "kube-api-access-bd648". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:56:21.982896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:21.982822    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd648\" (UniqueName: \"kubernetes.io/projected/e77e5a58-b702-46ec-a2ae-e8a8603e5a1d-kube-api-access-bd648\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:21.982896 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:21.982847    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77e5a58-b702-46ec-a2ae-e8a8603e5a1d-plugins-config-volume\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:22.060355 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:22.060327    2570 generic.go:356] "Generic (PLEG): container finished" podID="e77e5a58-b702-46ec-a2ae-e8a8603e5a1d" containerID="f80349a10e09c5dfd8cde2be2105ab8524adba3a0af3e495115459c07a29d689" exitCode=0
Sep 09 10:56:22.060500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:22.060375    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"
Sep 09 10:56:22.060500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:22.060406    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2" event={"ID":"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d","Type":"ContainerDied","Data":"f80349a10e09c5dfd8cde2be2105ab8524adba3a0af3e495115459c07a29d689"}
Sep 09 10:56:22.060500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:22.060454    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2" event={"ID":"e77e5a58-b702-46ec-a2ae-e8a8603e5a1d","Type":"ContainerDied","Data":"182a7f798b4c7e398d9d1e43b3a56e6188bae5b4743e7836b3f2119a99dcf632"}
Sep 09 10:56:22.060500 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:22.060477    2570 scope.go:117] "RemoveContainer" containerID="f80349a10e09c5dfd8cde2be2105ab8524adba3a0af3e495115459c07a29d689"
Sep 09 10:56:22.069348 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:22.069332    2570 scope.go:117] "RemoveContainer" containerID="f80349a10e09c5dfd8cde2be2105ab8524adba3a0af3e495115459c07a29d689"
Sep 09 10:56:22.069591 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:56:22.069573    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80349a10e09c5dfd8cde2be2105ab8524adba3a0af3e495115459c07a29d689\": container with ID starting with f80349a10e09c5dfd8cde2be2105ab8524adba3a0af3e495115459c07a29d689 not found: ID does not exist" containerID="f80349a10e09c5dfd8cde2be2105ab8524adba3a0af3e495115459c07a29d689"
Sep 09 10:56:22.069648 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:22.069603    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80349a10e09c5dfd8cde2be2105ab8524adba3a0af3e495115459c07a29d689"} err="failed to get container status \"f80349a10e09c5dfd8cde2be2105ab8524adba3a0af3e495115459c07a29d689\": rpc error: code = NotFound desc = could not find container \"f80349a10e09c5dfd8cde2be2105ab8524adba3a0af3e495115459c07a29d689\": container with ID starting with f80349a10e09c5dfd8cde2be2105ab8524adba3a0af3e495115459c07a29d689 not found: ID does not exist"
Sep 09 10:56:22.079692 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:22.079668    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:22.085656 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:22.085634    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"]
Sep 09 10:56:22.090265 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:22.090246    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-798524-ddfcf6f86-t7rt2"]
Sep 09 10:56:22.846865 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:22.846827    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e77e5a58-b702-46ec-a2ae-e8a8603e5a1d" path="/var/lib/kubelet/pods/e77e5a58-b702-46ec-a2ae-e8a8603e5a1d/volumes"
Sep 09 10:56:28.588171 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:28.588132    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:28.692246 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:28.692212    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:28.860407 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:28.860327    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:29.471170 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:29.471137    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:36.026572 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:36.026534    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:36.987637 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:36.987606    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"]
Sep 09 10:56:37.005615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.005584    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:37.539952 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.539919    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:37.874731 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.874651    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"]
Sep 09 10:56:37.875080 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.875068    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6138ae9-1c05-4761-a1a9-5f24c2d49f49" containerName="maas-api"
Sep 09 10:56:37.875134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.875083    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6138ae9-1c05-4761-a1a9-5f24c2d49f49" containerName="maas-api"
Sep 09 10:56:37.875134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.875096    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy"
Sep 09 10:56:37.875134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.875102    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy"
Sep 09 10:56:37.875134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.875116    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy"
Sep 09 10:56:37.875134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.875122    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy"
Sep 09 10:56:37.875134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.875131    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e77e5a58-b702-46ec-a2ae-e8a8603e5a1d" containerName="payload-pre-processing"
Sep 09 10:56:37.875134 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.875136    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e77e5a58-b702-46ec-a2ae-e8a8603e5a1d" containerName="payload-pre-processing"
Sep 09 10:56:37.875341 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.875196    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy"
Sep 09 10:56:37.875341 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.875205    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6138ae9-1c05-4761-a1a9-5f24c2d49f49" containerName="maas-api"
Sep 09 10:56:37.875341 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.875211    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ccaa45d-62c6-4046-95e6-80885aa208d5" containerName="istio-proxy"
Sep 09 10:56:37.875341 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.875216    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e77e5a58-b702-46ec-a2ae-e8a8603e5a1d" containerName="payload-pre-processing"
Sep 09 10:56:37.878264 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.878242    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:37.880948 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.880927    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-33d685-openshift-default-dockercfg-k2v65\""
Sep 09 10:56:37.893795 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.893738    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"]
Sep 09 10:56:37.896195 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.896173    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:37.919667 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.919633    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-envoy\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:37.919818 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.919672    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8jsm\" (UniqueName: \"kubernetes.io/projected/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-kube-api-access-h8jsm\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:37.919925 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.919895    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-workload-certs\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:37.920122 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.920101    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-data\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:37.920190 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.920154    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-workload-socket\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:37.920280 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.920244    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-token\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:37.920344 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.920297    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-podinfo\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:37.920420 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.920402    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istiod-ca-cert\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:37.920475 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:37.920435    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-credential-socket\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.021991 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.021959    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-workload-socket\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.022191 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.022004    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-token\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.022191 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.022022    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-podinfo\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.022191 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.022146    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istiod-ca-cert\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.022191 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.022185    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-credential-socket\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.022425 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.022280    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-envoy\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.022425 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.022308    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8jsm\" (UniqueName: \"kubernetes.io/projected/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-kube-api-access-h8jsm\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.022425 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.022333    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-workload-certs\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.022425 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.022375    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-data\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.022621 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.022504    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-workload-socket\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.022673 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.022617    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-credential-socket\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.022821 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.022803    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-workload-certs\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.022881 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.022823    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-data\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.022919 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.022902    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istiod-ca-cert\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.024432 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.024409    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-envoy\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.024565 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.024548    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-podinfo\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.030503 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.030475    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8jsm\" (UniqueName: \"kubernetes.io/projected/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-kube-api-access-h8jsm\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.030595 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.030525    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-token\") pod \"e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") " pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.188782 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.188690    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:38.461895 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.461872    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"]
Sep 09 10:56:38.463661 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:56:38.463634    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfa1f0c_b258_439b_b9b0_4341bdb09aa4.slice/crio-a092b35acd1628b497b29b6793ec835a8caf223eb83a134cb867f2e45087db6b WatchSource:0}: Error finding container a092b35acd1628b497b29b6793ec835a8caf223eb83a134cb867f2e45087db6b: Status 404 returned error can't find the container with id a092b35acd1628b497b29b6793ec835a8caf223eb83a134cb867f2e45087db6b
Sep 09 10:56:38.465830 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.465797    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:56:38.465927 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.465874    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:56:38.465927 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.465918    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:56:38.477052 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:38.477030    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:39.146119 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:39.146085    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2" event={"ID":"abfa1f0c-b258-439b-b9b0-4341bdb09aa4","Type":"ContainerStarted","Data":"f5564fa869ad101dc842f1a155c34ca1b03b8c5b87105307f21ecf2b1b138b6c"}
Sep 09 10:56:39.146473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:39.146136    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2" event={"ID":"abfa1f0c-b258-439b-b9b0-4341bdb09aa4","Type":"ContainerStarted","Data":"a092b35acd1628b497b29b6793ec835a8caf223eb83a134cb867f2e45087db6b"}
Sep 09 10:56:39.171126 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:39.171070    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2" podStartSLOduration=2.171055353 podStartE2EDuration="2.171055353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:56:39.169086112 +0000 UTC m=+1838.832725086" watchObservedRunningTime="2026-09-09 10:56:39.171055353 +0000 UTC m=+1838.834694386"
Sep 09 10:56:39.189567 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:39.189539    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:39.194692 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:39.194670    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:39.918266 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:39.918222    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"]
Sep 09 10:56:39.918492 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:39.918472    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265" podUID="0d5434d4-fd82-4f46-b5a3-15aea93fecb3" containerName="maas-api" containerID="cri-o://1b361f3b5237214bc95ca9ac5f8664852e707c38026098b841203569153aabed" gracePeriod=30
Sep 09 10:56:39.945174 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:39.945146    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:39.984818 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:39.984749    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"]
Sep 09 10:56:39.985072 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:39.985030    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj" podUID="61c8b2f1-b7f7-4a32-a450-0afb0b9345ba" containerName="payload-processing" containerID="cri-o://1a84f7b2b40b5def98b2b01b77ca83f42be7efd62b9bdb84cd1574294a3792d6" gracePeriod=30
Sep 09 10:56:40.049687 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.049650    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:40.153490 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.153377    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:40.156557 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.156522    2570 generic.go:356] "Generic (PLEG): container finished" podID="61c8b2f1-b7f7-4a32-a450-0afb0b9345ba" containerID="1a84f7b2b40b5def98b2b01b77ca83f42be7efd62b9bdb84cd1574294a3792d6" exitCode=0
Sep 09 10:56:40.156686 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.156619    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj" event={"ID":"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba","Type":"ContainerDied","Data":"1a84f7b2b40b5def98b2b01b77ca83f42be7efd62b9bdb84cd1574294a3792d6"}
Sep 09 10:56:40.159325 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.158489    2570 generic.go:356] "Generic (PLEG): container finished" podID="0d5434d4-fd82-4f46-b5a3-15aea93fecb3" containerID="1b361f3b5237214bc95ca9ac5f8664852e707c38026098b841203569153aabed" exitCode=0
Sep 09 10:56:40.159325 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.158538    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265" event={"ID":"0d5434d4-fd82-4f46-b5a3-15aea93fecb3","Type":"ContainerDied","Data":"1b361f3b5237214bc95ca9ac5f8664852e707c38026098b841203569153aabed"}
Sep 09 10:56:40.159518 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.159344    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:40.161556 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.161532    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:56:40.250590 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.250562    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:56:40.258376 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.258349    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:40.294343 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.294321    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"
Sep 09 10:56:40.346870 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.346837    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c8b2f1-b7f7-4a32-a450-0afb0b9345ba-plugins-config-volume\") pod \"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba\" (UID: \"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba\") "
Sep 09 10:56:40.347028 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.346894    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dwxn\" (UniqueName: \"kubernetes.io/projected/61c8b2f1-b7f7-4a32-a450-0afb0b9345ba-kube-api-access-2dwxn\") pod \"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba\" (UID: \"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba\") "
Sep 09 10:56:40.347096 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.347067    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bjvm\" (UniqueName: \"kubernetes.io/projected/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-kube-api-access-6bjvm\") pod \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") "
Sep 09 10:56:40.347191 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.347173    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-openshift-service-ca\") pod \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") "
Sep 09 10:56:40.347255 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.347238    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c8b2f1-b7f7-4a32-a450-0afb0b9345ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "61c8b2f1-b7f7-4a32-a450-0afb0b9345ba" (UID: "61c8b2f1-b7f7-4a32-a450-0afb0b9345ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:56:40.347255 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.347245    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-maas-api-tls\") pod \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") "
Sep 09 10:56:40.347329 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.347282    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-metrics-certs\") pod \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\" (UID: \"0d5434d4-fd82-4f46-b5a3-15aea93fecb3\") "
Sep 09 10:56:40.347494 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.347473    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d5434d4-fd82-4f46-b5a3-15aea93fecb3" (UID: "0d5434d4-fd82-4f46-b5a3-15aea93fecb3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:56:40.347675 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.347658    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-openshift-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:40.347745 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.347676    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c8b2f1-b7f7-4a32-a450-0afb0b9345ba-plugins-config-volume\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:40.349320 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.349298    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0d5434d4-fd82-4f46-b5a3-15aea93fecb3" (UID: "0d5434d4-fd82-4f46-b5a3-15aea93fecb3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:56:40.349418 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.349344    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d5434d4-fd82-4f46-b5a3-15aea93fecb3" (UID: "0d5434d4-fd82-4f46-b5a3-15aea93fecb3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:56:40.349486 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.349470    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c8b2f1-b7f7-4a32-a450-0afb0b9345ba-kube-api-access-2dwxn" (OuterVolumeSpecName: "kube-api-access-2dwxn") pod "61c8b2f1-b7f7-4a32-a450-0afb0b9345ba" (UID: "61c8b2f1-b7f7-4a32-a450-0afb0b9345ba"). InnerVolumeSpecName "kube-api-access-2dwxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:56:40.349543 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.349493    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-kube-api-access-6bjvm" (OuterVolumeSpecName: "kube-api-access-6bjvm") pod "0d5434d4-fd82-4f46-b5a3-15aea93fecb3" (UID: "0d5434d4-fd82-4f46-b5a3-15aea93fecb3"). InnerVolumeSpecName "kube-api-access-6bjvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:56:40.449148 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.449069    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dwxn\" (UniqueName: \"kubernetes.io/projected/61c8b2f1-b7f7-4a32-a450-0afb0b9345ba-kube-api-access-2dwxn\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:40.449148 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.449094    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bjvm\" (UniqueName: \"kubernetes.io/projected/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-kube-api-access-6bjvm\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:40.449148 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.449105    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-maas-api-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:40.449148 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:40.449113    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d5434d4-fd82-4f46-b5a3-15aea93fecb3-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:41.164142 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:41.164066    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"
Sep 09 10:56:41.164142 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:41.164088    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj" event={"ID":"61c8b2f1-b7f7-4a32-a450-0afb0b9345ba","Type":"ContainerDied","Data":"996413ed185f23d1eeb8a746c455e670aca57dfbab46980248f8ec8f478a6a80"}
Sep 09 10:56:41.164142 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:41.164143    2570 scope.go:117] "RemoveContainer" containerID="1a84f7b2b40b5def98b2b01b77ca83f42be7efd62b9bdb84cd1574294a3792d6"
Sep 09 10:56:41.165538 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:41.165519    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"
Sep 09 10:56:41.165662 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:41.165568    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265" event={"ID":"0d5434d4-fd82-4f46-b5a3-15aea93fecb3","Type":"ContainerDied","Data":"3bc9ecbc932bd7b6f55ae552e12f4de9778981d85a9429a52947fe23fc4f8070"}
Sep 09 10:56:41.172788 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:41.172750    2570 scope.go:117] "RemoveContainer" containerID="1b361f3b5237214bc95ca9ac5f8664852e707c38026098b841203569153aabed"
Sep 09 10:56:41.184084 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:41.184065    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"]
Sep 09 10:56:41.190592 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:41.190573    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-62f893-748f67457c-d6clj"]
Sep 09 10:56:41.213691 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:41.213665    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"]
Sep 09 10:56:41.228185 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:41.228164    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-62f893-579fdb8f6d-d6265"]
Sep 09 10:56:41.665454 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:41.665410    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:42.453474 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.453442    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"]
Sep 09 10:56:42.454021 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.454002    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5434d4-fd82-4f46-b5a3-15aea93fecb3" containerName="maas-api"
Sep 09 10:56:42.454117 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.454023    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5434d4-fd82-4f46-b5a3-15aea93fecb3" containerName="maas-api"
Sep 09 10:56:42.454117 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.454037    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61c8b2f1-b7f7-4a32-a450-0afb0b9345ba" containerName="payload-processing"
Sep 09 10:56:42.454117 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.454045    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="61c8b2f1-b7f7-4a32-a450-0afb0b9345ba" containerName="payload-processing"
Sep 09 10:56:42.454262 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.454182    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="61c8b2f1-b7f7-4a32-a450-0afb0b9345ba" containerName="payload-processing"
Sep 09 10:56:42.454262 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.454196    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d5434d4-fd82-4f46-b5a3-15aea93fecb3" containerName="maas-api"
Sep 09 10:56:42.458942 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.458919    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:42.461689 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.461669    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-33d685-serving-cert\""
Sep 09 10:56:42.470814 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.470758    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"]
Sep 09 10:56:42.499994 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.499956    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:42.566429 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.566395    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd59a316-08c5-4d29-9eb6-6400a834c8dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:42.566587 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.566444    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nnsb\" (UniqueName: \"kubernetes.io/projected/fd59a316-08c5-4d29-9eb6-6400a834c8dc-kube-api-access-7nnsb\") pod \"maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:42.566587 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.566522    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-metrics-certs\") pod \"maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:42.566659 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.566596    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:42.598063 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.598025    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:42.667179 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.667151    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:42.667372 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.667207    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd59a316-08c5-4d29-9eb6-6400a834c8dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:42.667372 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.667244    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nnsb\" (UniqueName: \"kubernetes.io/projected/fd59a316-08c5-4d29-9eb6-6400a834c8dc-kube-api-access-7nnsb\") pod \"maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:42.667372 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.667277    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-metrics-certs\") pod \"maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:42.667372 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:56:42.667292    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-serving-cert: secret "maas-api-e2e-ipp-a-33d685-serving-cert" not found
Sep 09 10:56:42.667372 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:56:42.667355    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-maas-api-tls podName:fd59a316-08c5-4d29-9eb6-6400a834c8dc nodeName:}" failed. No retries permitted until 2026-09-09 10:56:43.167338775 +0000 UTC m=+1842.830977726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-maas-api-tls") pod "maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d" (UID: "fd59a316-08c5-4d29-9eb6-6400a834c8dc") : secret "maas-api-e2e-ipp-a-33d685-serving-cert" not found
Sep 09 10:56:42.668034 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.668015    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd59a316-08c5-4d29-9eb6-6400a834c8dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:42.669676 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.669660    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-metrics-certs\") pod \"maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:42.676399 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.676374    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nnsb\" (UniqueName: \"kubernetes.io/projected/fd59a316-08c5-4d29-9eb6-6400a834c8dc-kube-api-access-7nnsb\") pod \"maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:42.767449 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.767411    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:42.839822 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.839759    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d5434d4-fd82-4f46-b5a3-15aea93fecb3" path="/var/lib/kubelet/pods/0d5434d4-fd82-4f46-b5a3-15aea93fecb3/volumes"
Sep 09 10:56:42.840230 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:42.840213    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61c8b2f1-b7f7-4a32-a450-0afb0b9345ba" path="/var/lib/kubelet/pods/61c8b2f1-b7f7-4a32-a450-0afb0b9345ba/volumes"
Sep 09 10:56:43.023000 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.022922    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:43.124726 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.124697    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:43.171354 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.171326    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:43.174052 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.174021    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:43.215474 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.215447    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"]
Sep 09 10:56:43.220236 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.220218    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"
Sep 09 10:56:43.224926 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.224904    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-33d685-dockercfg-hln7r\""
Sep 09 10:56:43.226045 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.226027    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-33d685\""
Sep 09 10:56:43.235780 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.235749    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"]
Sep 09 10:56:43.272830 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.272805    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drsgl\" (UniqueName: \"kubernetes.io/projected/9bc39842-9218-4d3c-8123-0e4788203dc1-kube-api-access-drsgl\") pod \"payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf\" (UID: \"9bc39842-9218-4d3c-8123-0e4788203dc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"
Sep 09 10:56:43.272946 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.272894    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc39842-9218-4d3c-8123-0e4788203dc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf\" (UID: \"9bc39842-9218-4d3c-8123-0e4788203dc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"
Sep 09 10:56:43.370965 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.370596    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:43.377333 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.373813    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drsgl\" (UniqueName: \"kubernetes.io/projected/9bc39842-9218-4d3c-8123-0e4788203dc1-kube-api-access-drsgl\") pod \"payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf\" (UID: \"9bc39842-9218-4d3c-8123-0e4788203dc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"
Sep 09 10:56:43.377333 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.373927    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc39842-9218-4d3c-8123-0e4788203dc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf\" (UID: \"9bc39842-9218-4d3c-8123-0e4788203dc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"
Sep 09 10:56:43.377333 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.374866    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc39842-9218-4d3c-8123-0e4788203dc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf\" (UID: \"9bc39842-9218-4d3c-8123-0e4788203dc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"
Sep 09 10:56:43.390461 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.390430    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drsgl\" (UniqueName: \"kubernetes.io/projected/9bc39842-9218-4d3c-8123-0e4788203dc1-kube-api-access-drsgl\") pod \"payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf\" (UID: \"9bc39842-9218-4d3c-8123-0e4788203dc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"
Sep 09 10:56:43.532303 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.532271    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"
Sep 09 10:56:43.658425 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.658401    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"]
Sep 09 10:56:43.660127 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:56:43.660102    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd59a316_08c5_4d29_9eb6_6400a834c8dc.slice/crio-e084555f75a7f4ac4b1d0a0b9ead099958b9ef24b5dac2f161a103f5115bea7b WatchSource:0}: Error finding container e084555f75a7f4ac4b1d0a0b9ead099958b9ef24b5dac2f161a103f5115bea7b: Status 404 returned error can't find the container with id e084555f75a7f4ac4b1d0a0b9ead099958b9ef24b5dac2f161a103f5115bea7b
Sep 09 10:56:43.706023 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.705997    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:43.804573 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:43.804551    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"]
Sep 09 10:56:43.806086 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:56:43.806059    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc39842_9218_4d3c_8123_0e4788203dc1.slice/crio-9057fbe5eb154170558cde145a1c2f47c3ef9031622606bb1eff4b5a51e8a2f2 WatchSource:0}: Error finding container 9057fbe5eb154170558cde145a1c2f47c3ef9031622606bb1eff4b5a51e8a2f2: Status 404 returned error can't find the container with id 9057fbe5eb154170558cde145a1c2f47c3ef9031622606bb1eff4b5a51e8a2f2
Sep 09 10:56:44.185557 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:44.185517    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf" event={"ID":"9bc39842-9218-4d3c-8123-0e4788203dc1","Type":"ContainerStarted","Data":"738aeef850c0249caa352dcbb676d97bb548ae4fa2e691740585fd83937f097c"}
Sep 09 10:56:44.185737 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:44.185573    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf" event={"ID":"9bc39842-9218-4d3c-8123-0e4788203dc1","Type":"ContainerStarted","Data":"9057fbe5eb154170558cde145a1c2f47c3ef9031622606bb1eff4b5a51e8a2f2"}
Sep 09 10:56:44.185737 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:44.185610    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"
Sep 09 10:56:44.187152 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:44.187130    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d" event={"ID":"fd59a316-08c5-4d29-9eb6-6400a834c8dc","Type":"ContainerStarted","Data":"965bba994e88d9d96df849a52815e30729b65275b28442bbac5b95379df99914"}
Sep 09 10:56:44.187260 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:44.187172    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d" event={"ID":"fd59a316-08c5-4d29-9eb6-6400a834c8dc","Type":"ContainerStarted","Data":"e084555f75a7f4ac4b1d0a0b9ead099958b9ef24b5dac2f161a103f5115bea7b"}
Sep 09 10:56:44.187260 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:44.187200    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:44.207594 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:44.207548    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf" podStartSLOduration=1.207533399 podStartE2EDuration="1.207533399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:56:44.205832431 +0000 UTC m=+1843.869471406" watchObservedRunningTime="2026-09-09 10:56:44.207533399 +0000 UTC m=+1843.871172372"
Sep 09 10:56:44.560071 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:44.560042    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:47.059311 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:47.059277    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:47.154231 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:47.154199    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:47.336186 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:47.336111    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:47.927826 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:47.927791    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:50.197937 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:50.197911    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:56:50.218264 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:50.218220    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d" podStartSLOduration=7.867113269 podStartE2EDuration="8.218207604s" totalImagesPullingTime="351.094335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:56:44.223931866 +0000 UTC m=+1843.887570840" watchObservedRunningTime="2026-09-09 10:56:50.218207604 +0000 UTC m=+1849.881846577"
Sep 09 10:56:52.006607 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.006568    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9" podUID="8701c907-c853-4ce7-981d-37c36aa9c4d0" containerName="main" containerID="cri-o://016daa171861841d020db3bbfb6ff4b080ebb0c09c6e675f001c8a1a8934d766" gracePeriod=45
Sep 09 10:56:52.231360 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.231332    2570 generic.go:356] "Generic (PLEG): container finished" podID="8701c907-c853-4ce7-981d-37c36aa9c4d0" containerID="016daa171861841d020db3bbfb6ff4b080ebb0c09c6e675f001c8a1a8934d766" exitCode=0
Sep 09 10:56:52.231508 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.231405    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9" event={"ID":"8701c907-c853-4ce7-981d-37c36aa9c4d0","Type":"ContainerDied","Data":"016daa171861841d020db3bbfb6ff4b080ebb0c09c6e675f001c8a1a8934d766"}
Sep 09 10:56:52.291138 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.291115    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:56:52.466305 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.466266    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-tmp-dir\") pod \"8701c907-c853-4ce7-981d-37c36aa9c4d0\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") "
Sep 09 10:56:52.466509 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.466339    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l986s\" (UniqueName: \"kubernetes.io/projected/8701c907-c853-4ce7-981d-37c36aa9c4d0-kube-api-access-l986s\") pod \"8701c907-c853-4ce7-981d-37c36aa9c4d0\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") "
Sep 09 10:56:52.466509 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.466391    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8701c907-c853-4ce7-981d-37c36aa9c4d0-tls-certs\") pod \"8701c907-c853-4ce7-981d-37c36aa9c4d0\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") "
Sep 09 10:56:52.466509 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.466427    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-dshm\") pod \"8701c907-c853-4ce7-981d-37c36aa9c4d0\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") "
Sep 09 10:56:52.466509 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.466473    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-home\") pod \"8701c907-c853-4ce7-981d-37c36aa9c4d0\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") "
Sep 09 10:56:52.466509 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.466494    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8701c907-c853-4ce7-981d-37c36aa9c4d0" (UID: "8701c907-c853-4ce7-981d-37c36aa9c4d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:52.466809 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.466514    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-model-cache\") pod \"8701c907-c853-4ce7-981d-37c36aa9c4d0\" (UID: \"8701c907-c853-4ce7-981d-37c36aa9c4d0\") "
Sep 09 10:56:52.466809 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.466615    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-home" (OuterVolumeSpecName: "home") pod "8701c907-c853-4ce7-981d-37c36aa9c4d0" (UID: "8701c907-c853-4ce7-981d-37c36aa9c4d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:52.466809 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.466711    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "8701c907-c853-4ce7-981d-37c36aa9c4d0" (UID: "8701c907-c853-4ce7-981d-37c36aa9c4d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:52.467176 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.467156    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-home\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:52.467257 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.467178    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-model-cache\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:52.467257 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.467188    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-tmp-dir\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:52.469058 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.469029    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8701c907-c853-4ce7-981d-37c36aa9c4d0-kube-api-access-l986s" (OuterVolumeSpecName: "kube-api-access-l986s") pod "8701c907-c853-4ce7-981d-37c36aa9c4d0" (UID: "8701c907-c853-4ce7-981d-37c36aa9c4d0"). InnerVolumeSpecName "kube-api-access-l986s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:56:52.469154 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.469118    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8701c907-c853-4ce7-981d-37c36aa9c4d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8701c907-c853-4ce7-981d-37c36aa9c4d0" (UID: "8701c907-c853-4ce7-981d-37c36aa9c4d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:56:52.469154 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.469119    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-dshm" (OuterVolumeSpecName: "dshm") pod "8701c907-c853-4ce7-981d-37c36aa9c4d0" (UID: "8701c907-c853-4ce7-981d-37c36aa9c4d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:52.568312 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.568223    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l986s\" (UniqueName: \"kubernetes.io/projected/8701c907-c853-4ce7-981d-37c36aa9c4d0-kube-api-access-l986s\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:52.568312 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.568253    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8701c907-c853-4ce7-981d-37c36aa9c4d0-tls-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:52.568312 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:52.568263    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8701c907-c853-4ce7-981d-37c36aa9c4d0-dshm\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:56:53.238700 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:53.238667    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"
Sep 09 10:56:53.238700 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:53.238689    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9" event={"ID":"8701c907-c853-4ce7-981d-37c36aa9c4d0","Type":"ContainerDied","Data":"3602f0107489298dbad5e28573b2ac8a4bc2ed98a7552f8c4130e62715db7344"}
Sep 09 10:56:53.239251 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:53.238734    2570 scope.go:117] "RemoveContainer" containerID="016daa171861841d020db3bbfb6ff4b080ebb0c09c6e675f001c8a1a8934d766"
Sep 09 10:56:53.263787 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:53.262780    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"]
Sep 09 10:56:53.266646 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:53.266619    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/rate-test-model-535b75-kserve-5b4697c87f-rlcb9"]
Sep 09 10:56:54.839539 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:54.839506    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8701c907-c853-4ce7-981d-37c36aa9c4d0" path="/var/lib/kubelet/pods/8701c907-c853-4ce7-981d-37c36aa9c4d0/volumes"
Sep 09 10:56:55.194306 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:55.194221    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"
Sep 09 10:56:58.318283 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:58.318249    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:58.418648 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:58.418594    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:58.517961 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:58.517928    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:58.618657 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:58.618583    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:56:58.919471 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:56:58.919391    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:00.293537 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:00.293475    2570 generic.go:356] "Generic (PLEG): container finished" podID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerID="f5564fa869ad101dc842f1a155c34ca1b03b8c5b87105307f21ecf2b1b138b6c" exitCode=0
Sep 09 10:57:00.293938 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:00.293548    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2" event={"ID":"abfa1f0c-b258-439b-b9b0-4341bdb09aa4","Type":"ContainerDied","Data":"f5564fa869ad101dc842f1a155c34ca1b03b8c5b87105307f21ecf2b1b138b6c"}
Sep 09 10:57:00.294005 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:00.293990    2570 scope.go:117] "RemoveContainer" containerID="f5564fa869ad101dc842f1a155c34ca1b03b8c5b87105307f21ecf2b1b138b6c"
Sep 09 10:57:00.295048 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:00.295015    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:57:00.295111 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:00.295097    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:57:00.295148 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:00.295131    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:57:01.299453 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:01.299417    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2" event={"ID":"abfa1f0c-b258-439b-b9b0-4341bdb09aa4","Type":"ContainerStarted","Data":"0d7a077cc678b87f2befabe9afe383190486921c7603c0a68aa159f2d1e0d486"}
Sep 09 10:57:02.188851 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:02.188812    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:57:02.189097 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:02.189073    2570 patch_prober.go:28] interesting pod/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 10:57:02.189167 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:02.189122    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2" podUID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 10:57:03.189734 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:03.189686    2570 patch_prober.go:28] interesting pod/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 10:57:03.190243 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:03.189758    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2" podUID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 10:57:03.530227 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:03.530180    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:04.189872 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:04.189831    2570 patch_prober.go:28] interesting pod/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 10:57:04.190348 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:04.189897    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2" podUID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 10:57:05.189104 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:05.189066    2570 patch_prober.go:28] interesting pod/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 10:57:05.189293 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:05.189163    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2" podUID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 10:57:05.579238 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:05.579205    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:05.681608 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:05.681571    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:05.867685 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:05.867585    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:06.189288 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:06.189200    2570 patch_prober.go:28] interesting pod/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 10:57:06.189288 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:06.189253    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2" podUID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 10:57:06.284957 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:06.284926    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:06.857695 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:06.857663    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:07.171154 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:07.171074    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:07.193551 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:07.193526    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:57:07.193815 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:07.193794    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:57:07.194553 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:07.194535    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 10:57:07.296110 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:07.296082    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:10.536583 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:10.536543    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:10.629171 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:10.629135    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:11.709480 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:11.709447    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:11.805004 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:11.804972    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:12.050051 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:12.050015    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:12.219219 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:12.219187    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:12.314979 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:12.314897    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:29.793908 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:29.793831    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:29.894696 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:29.894661    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:36.671983 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:36.671948    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:57.429649 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:57.429614    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:57:58.683926 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:57:58.683883    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:22.311398 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.311367    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:22.621195 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.621112    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"]
Sep 09 10:58:22.621530 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.621517    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8701c907-c853-4ce7-981d-37c36aa9c4d0" containerName="main"
Sep 09 10:58:22.621573 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.621532    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8701c907-c853-4ce7-981d-37c36aa9c4d0" containerName="main"
Sep 09 10:58:22.621625 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.621616    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8701c907-c853-4ce7-981d-37c36aa9c4d0" containerName="main"
Sep 09 10:58:22.624689 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.624671    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.627555 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.627533    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-f7ff8b-openshift-default-dockercfg-gcp6s\""
Sep 09 10:58:22.632529 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.632507    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-workload-socket\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.632615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.632556    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-data\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.632615 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.632598    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-podinfo\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.632707 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.632647    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6xw\" (UniqueName: \"kubernetes.io/projected/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-kube-api-access-vb6xw\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.632707 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.632680    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-workload-certs\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.632806 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.632734    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-token\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.632883 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.632865    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-envoy\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.632947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.632898    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.632947 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.632932    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-credential-socket\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.639272 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.639249    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"]
Sep 09 10:58:22.647667 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.647643    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:22.734364 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.734330    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-envoy\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.734566 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.734371    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.734566 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.734393    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-credential-socket\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.734566 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.734440    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-workload-socket\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.734566 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.734468    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-data\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.734566 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.734503    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-podinfo\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.734566 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.734548    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6xw\" (UniqueName: \"kubernetes.io/projected/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-kube-api-access-vb6xw\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.734931 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.734574    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-workload-certs\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.734931 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.734610    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-token\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.734931 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.734913    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-data\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.735090 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.734917    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-credential-socket\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.735090 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.734965    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-workload-socket\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.735194 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.735171    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-workload-certs\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.735332 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.735315    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.736829 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.736808    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-envoy\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.737098 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.737080    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-podinfo\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.743832 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.743812    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-token\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.744042 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.744022    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6xw\" (UniqueName: \"kubernetes.io/projected/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-kube-api-access-vb6xw\") pod \"e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") " pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:22.876002 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.875929    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:22.934757 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:22.934730    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:23.216816 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:23.213015    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"]
Sep 09 10:58:23.219932 ip-10-0-128-193 kubenswrapper[2570]: W0909 10:58:23.219895    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdcc01d_76ba_4ac4_aa33_c03307d3caca.slice/crio-3bef544fc6c9aa181008624e00244b426bef87e8c2e2eac008d8e714cbd1e1cd WatchSource:0}: Error finding container 3bef544fc6c9aa181008624e00244b426bef87e8c2e2eac008d8e714cbd1e1cd: Status 404 returned error can't find the container with id 3bef544fc6c9aa181008624e00244b426bef87e8c2e2eac008d8e714cbd1e1cd
Sep 09 10:58:23.221939 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:23.221902    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:58:23.222068 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:23.222003    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:58:23.222068 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:23.222051    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:58:23.717526 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:23.717490    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" event={"ID":"5cdcc01d-76ba-4ac4-aa33-c03307d3caca","Type":"ContainerStarted","Data":"57faa9b8b12dff959582bf746ea200fd6aa7dd5de51e395756bb839448f4d372"}
Sep 09 10:58:23.717955 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:23.717552    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" event={"ID":"5cdcc01d-76ba-4ac4-aa33-c03307d3caca","Type":"ContainerStarted","Data":"3bef544fc6c9aa181008624e00244b426bef87e8c2e2eac008d8e714cbd1e1cd"}
Sep 09 10:58:23.741125 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:23.741074    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" podStartSLOduration=1.741060914 podStartE2EDuration="1.741060914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:58:23.739176388 +0000 UTC m=+1943.402815365" watchObservedRunningTime="2026-09-09 10:58:23.741060914 +0000 UTC m=+1943.404699887"
Sep 09 10:58:23.935392 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:23.935349    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:24.941084 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:24.941053    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:25.728925 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:25.728897    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:25.729868 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:25.729851    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:25.748066 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:25.748041    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:25.846546 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:25.846518    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:27.053832 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:27.053802    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:27.149583 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:27.149553    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:27.620806 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:27.620751    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:27.710503 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:27.710460    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:27.801113 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:27.801060    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:46.834122 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:46.834086    2570 generic.go:356] "Generic (PLEG): container finished" podID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerID="57faa9b8b12dff959582bf746ea200fd6aa7dd5de51e395756bb839448f4d372" exitCode=0
Sep 09 10:58:46.841369 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:46.841340    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" event={"ID":"5cdcc01d-76ba-4ac4-aa33-c03307d3caca","Type":"ContainerDied","Data":"57faa9b8b12dff959582bf746ea200fd6aa7dd5de51e395756bb839448f4d372"}
Sep 09 10:58:46.841752 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:46.841736    2570 scope.go:117] "RemoveContainer" containerID="57faa9b8b12dff959582bf746ea200fd6aa7dd5de51e395756bb839448f4d372"
Sep 09 10:58:46.842912 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:46.842883    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:58:46.842999 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:46.842948    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:58:46.842999 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:46.842982    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:58:47.841729 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:47.840815    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" event={"ID":"5cdcc01d-76ba-4ac4-aa33-c03307d3caca","Type":"ContainerStarted","Data":"28f84825281744d75705e13fa67872c88e2b3e7ec57b5fb652c40c156efcbe0b"}
Sep 09 10:58:47.935259 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:47.935225    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:47.935436 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:47.935405    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 09 10:58:47.935485 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:47.935459    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 09 10:58:48.935741 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:48.935708    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 09 10:58:48.936128 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:48.935759    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 09 10:58:49.935246 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:49.935209    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 09 10:58:49.935424 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:49.935268    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 09 10:58:50.718190 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:50.718146    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:50.936073 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:50.936039    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 09 10:58:50.936242 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:50.936097    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 09 10:58:51.936056 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:51.936027    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 09 10:58:51.936457 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:51.936079    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 09 10:58:53.037185 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:53.037090    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:53.037652 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:53.037474    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:53.038338 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:53.038317    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:58:57.735456 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:57.735393    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:57.813396 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:57.813360    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:57.904820 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:57.904759    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:58.686015 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:58.685981    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:58:58.752677 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:58:58.752651    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:59:04.969457 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:04.969423    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:59:05.075533 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:05.075501    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:59:05.121051 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:05.121014    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"]
Sep 09 10:59:05.121268 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:05.121244    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerName="istio-proxy" containerID="cri-o://28f84825281744d75705e13fa67872c88e2b3e7ec57b5fb652c40c156efcbe0b" gracePeriod=30
Sep 09 10:59:05.149157 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:05.149129    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:59:08.039995 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:08.039962    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 10:59:08.040359 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:08.040021    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 10:59:09.414646 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:09.414614    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:59:10.389468 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.389409    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:59:10.490269 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.490235    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-podinfo\") pod \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") "
Sep 09 10:59:10.490715 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.490295    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb6xw\" (UniqueName: \"kubernetes.io/projected/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-kube-api-access-vb6xw\") pod \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") "
Sep 09 10:59:10.490715 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.490322    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-workload-certs\") pod \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") "
Sep 09 10:59:10.490715 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.490351    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istiod-ca-cert\") pod \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") "
Sep 09 10:59:10.490715 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.490411    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-envoy\") pod \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") "
Sep 09 10:59:10.490715 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.490477    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-data\") pod \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") "
Sep 09 10:59:10.490715 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.490550    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-workload-socket\") pod \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") "
Sep 09 10:59:10.490715 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.490621    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-credential-socket\") pod \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") "
Sep 09 10:59:10.490715 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.490647    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-token\") pod \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\" (UID: \"5cdcc01d-76ba-4ac4-aa33-c03307d3caca\") "
Sep 09 10:59:10.490715 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.490660    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5cdcc01d-76ba-4ac4-aa33-c03307d3caca" (UID: "5cdcc01d-76ba-4ac4-aa33-c03307d3caca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:59:10.491237 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.490760    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5cdcc01d-76ba-4ac4-aa33-c03307d3caca" (UID: "5cdcc01d-76ba-4ac4-aa33-c03307d3caca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:59:10.491237 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.490902    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5cdcc01d-76ba-4ac4-aa33-c03307d3caca" (UID: "5cdcc01d-76ba-4ac4-aa33-c03307d3caca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:59:10.491237 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.491101    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-data" (OuterVolumeSpecName: "istio-data") pod "5cdcc01d-76ba-4ac4-aa33-c03307d3caca" (UID: "5cdcc01d-76ba-4ac4-aa33-c03307d3caca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:59:10.491473 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.491325    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5cdcc01d-76ba-4ac4-aa33-c03307d3caca" (UID: "5cdcc01d-76ba-4ac4-aa33-c03307d3caca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:59:10.492131 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.492105    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-workload-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:10.492260 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.492136    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istiod-ca-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:10.492260 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.492150    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-data\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:10.492260 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.492164    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-workload-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:10.493158 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.492758    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5cdcc01d-76ba-4ac4-aa33-c03307d3caca" (UID: "5cdcc01d-76ba-4ac4-aa33-c03307d3caca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:59:10.493158 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.492791    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5cdcc01d-76ba-4ac4-aa33-c03307d3caca" (UID: "5cdcc01d-76ba-4ac4-aa33-c03307d3caca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:59:10.493158 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.493018    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-token" (OuterVolumeSpecName: "istio-token") pod "5cdcc01d-76ba-4ac4-aa33-c03307d3caca" (UID: "5cdcc01d-76ba-4ac4-aa33-c03307d3caca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:59:10.493353 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.493186    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-kube-api-access-vb6xw" (OuterVolumeSpecName: "kube-api-access-vb6xw") pod "5cdcc01d-76ba-4ac4-aa33-c03307d3caca" (UID: "5cdcc01d-76ba-4ac4-aa33-c03307d3caca"). InnerVolumeSpecName "kube-api-access-vb6xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:59:10.593619 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.593595    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-credential-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:10.593619 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.593617    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:10.593789 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.593627    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-podinfo\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:10.593789 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.593635    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb6xw\" (UniqueName: \"kubernetes.io/projected/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-kube-api-access-vb6xw\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:10.593789 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.593643    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdcc01d-76ba-4ac4-aa33-c03307d3caca-istio-envoy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:10.966618 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.966589    2570 generic.go:356] "Generic (PLEG): container finished" podID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerID="28f84825281744d75705e13fa67872c88e2b3e7ec57b5fb652c40c156efcbe0b" exitCode=0
Sep 09 10:59:10.966795 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.966630    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"
Sep 09 10:59:10.966795 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.966671    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" event={"ID":"5cdcc01d-76ba-4ac4-aa33-c03307d3caca","Type":"ContainerDied","Data":"28f84825281744d75705e13fa67872c88e2b3e7ec57b5fb652c40c156efcbe0b"}
Sep 09 10:59:10.966795 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.966730    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7" event={"ID":"5cdcc01d-76ba-4ac4-aa33-c03307d3caca","Type":"ContainerDied","Data":"3bef544fc6c9aa181008624e00244b426bef87e8c2e2eac008d8e714cbd1e1cd"}
Sep 09 10:59:10.966795 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.966758    2570 scope.go:117] "RemoveContainer" containerID="28f84825281744d75705e13fa67872c88e2b3e7ec57b5fb652c40c156efcbe0b"
Sep 09 10:59:10.975921 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.975906    2570 scope.go:117] "RemoveContainer" containerID="57faa9b8b12dff959582bf746ea200fd6aa7dd5de51e395756bb839448f4d372"
Sep 09 10:59:10.983601 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.983583    2570 scope.go:117] "RemoveContainer" containerID="28f84825281744d75705e13fa67872c88e2b3e7ec57b5fb652c40c156efcbe0b"
Sep 09 10:59:10.983887 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:59:10.983869    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f84825281744d75705e13fa67872c88e2b3e7ec57b5fb652c40c156efcbe0b\": container with ID starting with 28f84825281744d75705e13fa67872c88e2b3e7ec57b5fb652c40c156efcbe0b not found: ID does not exist" containerID="28f84825281744d75705e13fa67872c88e2b3e7ec57b5fb652c40c156efcbe0b"
Sep 09 10:59:10.983962 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.983899    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f84825281744d75705e13fa67872c88e2b3e7ec57b5fb652c40c156efcbe0b"} err="failed to get container status \"28f84825281744d75705e13fa67872c88e2b3e7ec57b5fb652c40c156efcbe0b\": rpc error: code = NotFound desc = could not find container \"28f84825281744d75705e13fa67872c88e2b3e7ec57b5fb652c40c156efcbe0b\": container with ID starting with 28f84825281744d75705e13fa67872c88e2b3e7ec57b5fb652c40c156efcbe0b not found: ID does not exist"
Sep 09 10:59:10.983962 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.983922    2570 scope.go:117] "RemoveContainer" containerID="57faa9b8b12dff959582bf746ea200fd6aa7dd5de51e395756bb839448f4d372"
Sep 09 10:59:10.984172 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:59:10.984149    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57faa9b8b12dff959582bf746ea200fd6aa7dd5de51e395756bb839448f4d372\": container with ID starting with 57faa9b8b12dff959582bf746ea200fd6aa7dd5de51e395756bb839448f4d372 not found: ID does not exist" containerID="57faa9b8b12dff959582bf746ea200fd6aa7dd5de51e395756bb839448f4d372"
Sep 09 10:59:10.984216 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.984178    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57faa9b8b12dff959582bf746ea200fd6aa7dd5de51e395756bb839448f4d372"} err="failed to get container status \"57faa9b8b12dff959582bf746ea200fd6aa7dd5de51e395756bb839448f4d372\": rpc error: code = NotFound desc = could not find container \"57faa9b8b12dff959582bf746ea200fd6aa7dd5de51e395756bb839448f4d372\": container with ID starting with 57faa9b8b12dff959582bf746ea200fd6aa7dd5de51e395756bb839448f4d372 not found: ID does not exist"
Sep 09 10:59:10.987840 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.987818    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"]
Sep 09 10:59:10.993128 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:10.993109    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f7ff8b-openshift-default-6dd6bc7fd-rlmf7"]
Sep 09 10:59:12.842294 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:12.842260    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" path="/var/lib/kubelet/pods/5cdcc01d-76ba-4ac4-aa33-c03307d3caca/volumes"
Sep 09 10:59:15.582132 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:15.582096    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:59:19.328483 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:19.328450    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:59:25.026591 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.026560    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"]
Sep 09 10:59:25.027098 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.026777    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d" podUID="fd59a316-08c5-4d29-9eb6-6400a834c8dc" containerName="maas-api" containerID="cri-o://965bba994e88d9d96df849a52815e30729b65275b28442bbac5b95379df99914" gracePeriod=30
Sep 09 10:59:25.096370 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.096331    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:59:25.101247 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.101220    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"]
Sep 09 10:59:25.101450 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.101426    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf" podUID="9bc39842-9218-4d3c-8123-0e4788203dc1" containerName="payload-pre-processing" containerID="cri-o://738aeef850c0249caa352dcbb676d97bb548ae4fa2e691740585fd83937f097c" gracePeriod=30
Sep 09 10:59:25.193163 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.193127    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf" podUID="9bc39842-9218-4d3c-8123-0e4788203dc1" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.75:9004: connect: connection refused"
Sep 09 10:59:25.193303 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.193131    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d" podUID="fd59a316-08c5-4d29-9eb6-6400a834c8dc" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.74:8443/health\": dial tcp 10.134.0.74:8443: connect: connection refused"
Sep 09 10:59:25.207566 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.207536    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:59:25.319326 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.319297    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:59:25.362842 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.362786    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:59:25.394880 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.394859    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"
Sep 09 10:59:25.424013 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.423966    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:59:25.426992 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.426973    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd59a316-08c5-4d29-9eb6-6400a834c8dc-openshift-service-ca\") pod \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") "
Sep 09 10:59:25.427079 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.427051    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-maas-api-tls\") pod \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") "
Sep 09 10:59:25.427138 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.427115    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-metrics-certs\") pod \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") "
Sep 09 10:59:25.427227 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.427209    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nnsb\" (UniqueName: \"kubernetes.io/projected/fd59a316-08c5-4d29-9eb6-6400a834c8dc-kube-api-access-7nnsb\") pod \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\" (UID: \"fd59a316-08c5-4d29-9eb6-6400a834c8dc\") "
Sep 09 10:59:25.427345 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.427321    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd59a316-08c5-4d29-9eb6-6400a834c8dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd59a316-08c5-4d29-9eb6-6400a834c8dc" (UID: "fd59a316-08c5-4d29-9eb6-6400a834c8dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:59:25.427560 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.427542    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd59a316-08c5-4d29-9eb6-6400a834c8dc-openshift-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:25.429352 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.429314    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd59a316-08c5-4d29-9eb6-6400a834c8dc" (UID: "fd59a316-08c5-4d29-9eb6-6400a834c8dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:59:25.429487 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.429312    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd59a316-08c5-4d29-9eb6-6400a834c8dc-kube-api-access-7nnsb" (OuterVolumeSpecName: "kube-api-access-7nnsb") pod "fd59a316-08c5-4d29-9eb6-6400a834c8dc" (UID: "fd59a316-08c5-4d29-9eb6-6400a834c8dc"). InnerVolumeSpecName "kube-api-access-7nnsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:59:25.429650 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.429623    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fd59a316-08c5-4d29-9eb6-6400a834c8dc" (UID: "fd59a316-08c5-4d29-9eb6-6400a834c8dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:59:25.528249 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.528213    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drsgl\" (UniqueName: \"kubernetes.io/projected/9bc39842-9218-4d3c-8123-0e4788203dc1-kube-api-access-drsgl\") pod \"9bc39842-9218-4d3c-8123-0e4788203dc1\" (UID: \"9bc39842-9218-4d3c-8123-0e4788203dc1\") "
Sep 09 10:59:25.528406 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.528313    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc39842-9218-4d3c-8123-0e4788203dc1-plugins-config-volume\") pod \"9bc39842-9218-4d3c-8123-0e4788203dc1\" (UID: \"9bc39842-9218-4d3c-8123-0e4788203dc1\") "
Sep 09 10:59:25.528621 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.528603    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nnsb\" (UniqueName: \"kubernetes.io/projected/fd59a316-08c5-4d29-9eb6-6400a834c8dc-kube-api-access-7nnsb\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:25.528685 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.528627    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-maas-api-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:25.528685 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.528640    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd59a316-08c5-4d29-9eb6-6400a834c8dc-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:25.528685 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.528620    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc39842-9218-4d3c-8123-0e4788203dc1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9bc39842-9218-4d3c-8123-0e4788203dc1" (UID: "9bc39842-9218-4d3c-8123-0e4788203dc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:59:25.530232 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.530211    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc39842-9218-4d3c-8123-0e4788203dc1-kube-api-access-drsgl" (OuterVolumeSpecName: "kube-api-access-drsgl") pod "9bc39842-9218-4d3c-8123-0e4788203dc1" (UID: "9bc39842-9218-4d3c-8123-0e4788203dc1"). InnerVolumeSpecName "kube-api-access-drsgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:59:25.629464 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.629399    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drsgl\" (UniqueName: \"kubernetes.io/projected/9bc39842-9218-4d3c-8123-0e4788203dc1-kube-api-access-drsgl\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:25.629464 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.629420    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc39842-9218-4d3c-8123-0e4788203dc1-plugins-config-volume\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 10:59:25.739584 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:25.739556    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:59:26.046991 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.046965    2570 generic.go:356] "Generic (PLEG): container finished" podID="9bc39842-9218-4d3c-8123-0e4788203dc1" containerID="738aeef850c0249caa352dcbb676d97bb548ae4fa2e691740585fd83937f097c" exitCode=0
Sep 09 10:59:26.047386 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.047011    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"
Sep 09 10:59:26.047386 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.047045    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf" event={"ID":"9bc39842-9218-4d3c-8123-0e4788203dc1","Type":"ContainerDied","Data":"738aeef850c0249caa352dcbb676d97bb548ae4fa2e691740585fd83937f097c"}
Sep 09 10:59:26.047386 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.047102    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf" event={"ID":"9bc39842-9218-4d3c-8123-0e4788203dc1","Type":"ContainerDied","Data":"9057fbe5eb154170558cde145a1c2f47c3ef9031622606bb1eff4b5a51e8a2f2"}
Sep 09 10:59:26.047386 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.047128    2570 scope.go:117] "RemoveContainer" containerID="738aeef850c0249caa352dcbb676d97bb548ae4fa2e691740585fd83937f097c"
Sep 09 10:59:26.048455 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.048428    2570 generic.go:356] "Generic (PLEG): container finished" podID="fd59a316-08c5-4d29-9eb6-6400a834c8dc" containerID="965bba994e88d9d96df849a52815e30729b65275b28442bbac5b95379df99914" exitCode=0
Sep 09 10:59:26.048529 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.048466    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"
Sep 09 10:59:26.048529 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.048513    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d" event={"ID":"fd59a316-08c5-4d29-9eb6-6400a834c8dc","Type":"ContainerDied","Data":"965bba994e88d9d96df849a52815e30729b65275b28442bbac5b95379df99914"}
Sep 09 10:59:26.048641 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.048561    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d" event={"ID":"fd59a316-08c5-4d29-9eb6-6400a834c8dc","Type":"ContainerDied","Data":"e084555f75a7f4ac4b1d0a0b9ead099958b9ef24b5dac2f161a103f5115bea7b"}
Sep 09 10:59:26.056882 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.056826    2570 scope.go:117] "RemoveContainer" containerID="738aeef850c0249caa352dcbb676d97bb548ae4fa2e691740585fd83937f097c"
Sep 09 10:59:26.057117 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:59:26.057096    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738aeef850c0249caa352dcbb676d97bb548ae4fa2e691740585fd83937f097c\": container with ID starting with 738aeef850c0249caa352dcbb676d97bb548ae4fa2e691740585fd83937f097c not found: ID does not exist" containerID="738aeef850c0249caa352dcbb676d97bb548ae4fa2e691740585fd83937f097c"
Sep 09 10:59:26.057170 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.057129    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738aeef850c0249caa352dcbb676d97bb548ae4fa2e691740585fd83937f097c"} err="failed to get container status \"738aeef850c0249caa352dcbb676d97bb548ae4fa2e691740585fd83937f097c\": rpc error: code = NotFound desc = could not find container \"738aeef850c0249caa352dcbb676d97bb548ae4fa2e691740585fd83937f097c\": container with ID starting with 738aeef850c0249caa352dcbb676d97bb548ae4fa2e691740585fd83937f097c not found: ID does not exist"
Sep 09 10:59:26.057170 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.057153    2570 scope.go:117] "RemoveContainer" containerID="965bba994e88d9d96df849a52815e30729b65275b28442bbac5b95379df99914"
Sep 09 10:59:26.065463 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.065444    2570 scope.go:117] "RemoveContainer" containerID="965bba994e88d9d96df849a52815e30729b65275b28442bbac5b95379df99914"
Sep 09 10:59:26.065715 ip-10-0-128-193 kubenswrapper[2570]: E0909 10:59:26.065699    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965bba994e88d9d96df849a52815e30729b65275b28442bbac5b95379df99914\": container with ID starting with 965bba994e88d9d96df849a52815e30729b65275b28442bbac5b95379df99914 not found: ID does not exist" containerID="965bba994e88d9d96df849a52815e30729b65275b28442bbac5b95379df99914"
Sep 09 10:59:26.065783 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.065720    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965bba994e88d9d96df849a52815e30729b65275b28442bbac5b95379df99914"} err="failed to get container status \"965bba994e88d9d96df849a52815e30729b65275b28442bbac5b95379df99914\": rpc error: code = NotFound desc = could not find container \"965bba994e88d9d96df849a52815e30729b65275b28442bbac5b95379df99914\": container with ID starting with 965bba994e88d9d96df849a52815e30729b65275b28442bbac5b95379df99914 not found: ID does not exist"
Sep 09 10:59:26.070936 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.070917    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"]
Sep 09 10:59:26.077005 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.076986    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33d685-58b9659b8b-cpwtf"]
Sep 09 10:59:26.089814 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.089792    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"]
Sep 09 10:59:26.093295 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.093278    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33d685-744db4fc45-rf76d"]
Sep 09 10:59:26.839315 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.839283    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bc39842-9218-4d3c-8123-0e4788203dc1" path="/var/lib/kubelet/pods/9bc39842-9218-4d3c-8123-0e4788203dc1/volumes"
Sep 09 10:59:26.839688 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:26.839675    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd59a316-08c5-4d29-9eb6-6400a834c8dc" path="/var/lib/kubelet/pods/fd59a316-08c5-4d29-9eb6-6400a834c8dc/volumes"
Sep 09 10:59:30.802504 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:30.802469    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 10:59:44.796487 ip-10-0-128-193 kubenswrapper[2570]: I0909 10:59:44.796450    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:00.721429 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:00.721397    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:06.855096 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:06.855058    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:08.390387 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:08.390352    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:14.705696 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:14.705659    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"]
Sep 09 11:00:14.706172 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:14.705937    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2" podUID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerName="istio-proxy" containerID="cri-o://0d7a077cc678b87f2befabe9afe383190486921c7603c0a68aa159f2d1e0d486" gracePeriod=30
Sep 09 11:00:14.743684 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:14.743658    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:14.895310 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:14.895277    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:15.815926 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:15.815871    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:19.963568 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:19.963543    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 11:00:20.132971 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.132945    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-token\") pod \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") "
Sep 09 11:00:20.133122 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.132994    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-envoy\") pod \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") "
Sep 09 11:00:20.133122 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133017    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-data\") pod \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") "
Sep 09 11:00:20.133122 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133051    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-credential-socket\") pod \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") "
Sep 09 11:00:20.133122 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133067    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-workload-certs\") pod \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") "
Sep 09 11:00:20.133122 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133100    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8jsm\" (UniqueName: \"kubernetes.io/projected/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-kube-api-access-h8jsm\") pod \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") "
Sep 09 11:00:20.133381 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133146    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-podinfo\") pod \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") "
Sep 09 11:00:20.133381 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133250    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-workload-socket\") pod \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") "
Sep 09 11:00:20.133381 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133303    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istiod-ca-cert\") pod \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\" (UID: \"abfa1f0c-b258-439b-b9b0-4341bdb09aa4\") "
Sep 09 11:00:20.133381 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133341    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-data" (OuterVolumeSpecName: "istio-data") pod "abfa1f0c-b258-439b-b9b0-4341bdb09aa4" (UID: "abfa1f0c-b258-439b-b9b0-4341bdb09aa4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:20.133592 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133447    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "abfa1f0c-b258-439b-b9b0-4341bdb09aa4" (UID: "abfa1f0c-b258-439b-b9b0-4341bdb09aa4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:20.133592 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133565    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "abfa1f0c-b258-439b-b9b0-4341bdb09aa4" (UID: "abfa1f0c-b258-439b-b9b0-4341bdb09aa4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:20.133704 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133672    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "abfa1f0c-b258-439b-b9b0-4341bdb09aa4" (UID: "abfa1f0c-b258-439b-b9b0-4341bdb09aa4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:20.133853 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133805    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-workload-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:20.133853 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133829    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-data\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:20.133853 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133845    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-credential-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:20.134094 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.133858    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-workload-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:20.134094 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.134001    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "abfa1f0c-b258-439b-b9b0-4341bdb09aa4" (UID: "abfa1f0c-b258-439b-b9b0-4341bdb09aa4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 11:00:20.135167 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.135148    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-kube-api-access-h8jsm" (OuterVolumeSpecName: "kube-api-access-h8jsm") pod "abfa1f0c-b258-439b-b9b0-4341bdb09aa4" (UID: "abfa1f0c-b258-439b-b9b0-4341bdb09aa4"). InnerVolumeSpecName "kube-api-access-h8jsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:00:20.135253 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.135185    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-token" (OuterVolumeSpecName: "istio-token") pod "abfa1f0c-b258-439b-b9b0-4341bdb09aa4" (UID: "abfa1f0c-b258-439b-b9b0-4341bdb09aa4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:00:20.135582 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.135561    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "abfa1f0c-b258-439b-b9b0-4341bdb09aa4" (UID: "abfa1f0c-b258-439b-b9b0-4341bdb09aa4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 11:00:20.135582 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.135567    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "abfa1f0c-b258-439b-b9b0-4341bdb09aa4" (UID: "abfa1f0c-b258-439b-b9b0-4341bdb09aa4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:20.234449 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.234425    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:20.234449 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.234446    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-envoy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:20.234575 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.234455    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8jsm\" (UniqueName: \"kubernetes.io/projected/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-kube-api-access-h8jsm\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:20.234575 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.234463    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istio-podinfo\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:20.234575 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.234471    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfa1f0c-b258-439b-b9b0-4341bdb09aa4-istiod-ca-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:20.317013 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.316988    2570 generic.go:356] "Generic (PLEG): container finished" podID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerID="0d7a077cc678b87f2befabe9afe383190486921c7603c0a68aa159f2d1e0d486" exitCode=0
Sep 09 11:00:20.317116 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.317034    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"
Sep 09 11:00:20.317116 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.317076    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2" event={"ID":"abfa1f0c-b258-439b-b9b0-4341bdb09aa4","Type":"ContainerDied","Data":"0d7a077cc678b87f2befabe9afe383190486921c7603c0a68aa159f2d1e0d486"}
Sep 09 11:00:20.317197 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.317126    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2" event={"ID":"abfa1f0c-b258-439b-b9b0-4341bdb09aa4","Type":"ContainerDied","Data":"a092b35acd1628b497b29b6793ec835a8caf223eb83a134cb867f2e45087db6b"}
Sep 09 11:00:20.317197 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.317149    2570 scope.go:117] "RemoveContainer" containerID="0d7a077cc678b87f2befabe9afe383190486921c7603c0a68aa159f2d1e0d486"
Sep 09 11:00:20.326440 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.326420    2570 scope.go:117] "RemoveContainer" containerID="f5564fa869ad101dc842f1a155c34ca1b03b8c5b87105307f21ecf2b1b138b6c"
Sep 09 11:00:20.333924 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.333905    2570 scope.go:117] "RemoveContainer" containerID="0d7a077cc678b87f2befabe9afe383190486921c7603c0a68aa159f2d1e0d486"
Sep 09 11:00:20.334206 ip-10-0-128-193 kubenswrapper[2570]: E0909 11:00:20.334186    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7a077cc678b87f2befabe9afe383190486921c7603c0a68aa159f2d1e0d486\": container with ID starting with 0d7a077cc678b87f2befabe9afe383190486921c7603c0a68aa159f2d1e0d486 not found: ID does not exist" containerID="0d7a077cc678b87f2befabe9afe383190486921c7603c0a68aa159f2d1e0d486"
Sep 09 11:00:20.334255 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.334220    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7a077cc678b87f2befabe9afe383190486921c7603c0a68aa159f2d1e0d486"} err="failed to get container status \"0d7a077cc678b87f2befabe9afe383190486921c7603c0a68aa159f2d1e0d486\": rpc error: code = NotFound desc = could not find container \"0d7a077cc678b87f2befabe9afe383190486921c7603c0a68aa159f2d1e0d486\": container with ID starting with 0d7a077cc678b87f2befabe9afe383190486921c7603c0a68aa159f2d1e0d486 not found: ID does not exist"
Sep 09 11:00:20.334255 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.334242    2570 scope.go:117] "RemoveContainer" containerID="f5564fa869ad101dc842f1a155c34ca1b03b8c5b87105307f21ecf2b1b138b6c"
Sep 09 11:00:20.334487 ip-10-0-128-193 kubenswrapper[2570]: E0909 11:00:20.334467    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5564fa869ad101dc842f1a155c34ca1b03b8c5b87105307f21ecf2b1b138b6c\": container with ID starting with f5564fa869ad101dc842f1a155c34ca1b03b8c5b87105307f21ecf2b1b138b6c not found: ID does not exist" containerID="f5564fa869ad101dc842f1a155c34ca1b03b8c5b87105307f21ecf2b1b138b6c"
Sep 09 11:00:20.334565 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.334490    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5564fa869ad101dc842f1a155c34ca1b03b8c5b87105307f21ecf2b1b138b6c"} err="failed to get container status \"f5564fa869ad101dc842f1a155c34ca1b03b8c5b87105307f21ecf2b1b138b6c\": rpc error: code = NotFound desc = could not find container \"f5564fa869ad101dc842f1a155c34ca1b03b8c5b87105307f21ecf2b1b138b6c\": container with ID starting with f5564fa869ad101dc842f1a155c34ca1b03b8c5b87105307f21ecf2b1b138b6c not found: ID does not exist"
Sep 09 11:00:20.340450 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.340430    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"]
Sep 09 11:00:20.344541 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.344522    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-33d685-openshift-default-cdc8b7fb8-2vmc2"]
Sep 09 11:00:20.841013 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:20.840981    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" path="/var/lib/kubelet/pods/abfa1f0c-b258-439b-b9b0-4341bdb09aa4/volumes"
Sep 09 11:00:25.748387 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:25.748283    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:25.845265 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:25.845228    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:26.012114 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:26.012035    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:27.214094 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:27.214060    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:27.274064 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:27.274035    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:33.551826 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:33.551794    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:33.651461 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:33.651405    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:43.877834 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:43.877796    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:43.950886 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:43.950857    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:44.052683 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:44.052655    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:44.115287 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:44.115256    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:44.474607 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:44.474578    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:50.815663 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:50.815628    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:51.100352 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:51.100269    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"]
Sep 09 11:00:51.100512 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:51.100488    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy" containerID="cri-o://a1502bb8307bb942cc3b8fb6b374413f28d5b8f07682eb6746393512c0d09121" gracePeriod=30
Sep 09 11:00:51.737841 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:51.737809    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:52.451885 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:52.451859    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:00:56.352370 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.352341    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 11:00:56.453882 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.453816    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1958cc9b-9cd5-4f98-b693-607d925c4be8-istiod-ca-cert\") pod \"1958cc9b-9cd5-4f98-b693-607d925c4be8\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") "
Sep 09 11:00:56.453882 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.453865    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7v7w\" (UniqueName: \"kubernetes.io/projected/1958cc9b-9cd5-4f98-b693-607d925c4be8-kube-api-access-b7v7w\") pod \"1958cc9b-9cd5-4f98-b693-607d925c4be8\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") "
Sep 09 11:00:56.454082 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.453913    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-credential-socket\") pod \"1958cc9b-9cd5-4f98-b693-607d925c4be8\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") "
Sep 09 11:00:56.454082 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.453935    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-token\") pod \"1958cc9b-9cd5-4f98-b693-607d925c4be8\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") "
Sep 09 11:00:56.454082 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.453973    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-workload-socket\") pod \"1958cc9b-9cd5-4f98-b693-607d925c4be8\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") "
Sep 09 11:00:56.454082 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454002    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-podinfo\") pod \"1958cc9b-9cd5-4f98-b693-607d925c4be8\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") "
Sep 09 11:00:56.454082 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454030    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-data\") pod \"1958cc9b-9cd5-4f98-b693-607d925c4be8\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") "
Sep 09 11:00:56.454346 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454090    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-envoy\") pod \"1958cc9b-9cd5-4f98-b693-607d925c4be8\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") "
Sep 09 11:00:56.454346 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454188    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1958cc9b-9cd5-4f98-b693-607d925c4be8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1958cc9b-9cd5-4f98-b693-607d925c4be8" (UID: "1958cc9b-9cd5-4f98-b693-607d925c4be8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 11:00:56.454346 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454213    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1958cc9b-9cd5-4f98-b693-607d925c4be8" (UID: "1958cc9b-9cd5-4f98-b693-607d925c4be8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:56.454346 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454264    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-workload-certs\") pod \"1958cc9b-9cd5-4f98-b693-607d925c4be8\" (UID: \"1958cc9b-9cd5-4f98-b693-607d925c4be8\") "
Sep 09 11:00:56.454346 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454339    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1958cc9b-9cd5-4f98-b693-607d925c4be8" (UID: "1958cc9b-9cd5-4f98-b693-607d925c4be8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:56.454615 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454499    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-data" (OuterVolumeSpecName: "istio-data") pod "1958cc9b-9cd5-4f98-b693-607d925c4be8" (UID: "1958cc9b-9cd5-4f98-b693-607d925c4be8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:56.454615 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454566    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1958cc9b-9cd5-4f98-b693-607d925c4be8" (UID: "1958cc9b-9cd5-4f98-b693-607d925c4be8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:56.454723 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454680    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-workload-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:56.454723 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454700    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1958cc9b-9cd5-4f98-b693-607d925c4be8-istiod-ca-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:56.454723 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454712    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-credential-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:56.454922 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454727    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-workload-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:56.454922 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.454736    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-data\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:56.456108 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.456081    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1958cc9b-9cd5-4f98-b693-607d925c4be8-kube-api-access-b7v7w" (OuterVolumeSpecName: "kube-api-access-b7v7w") pod "1958cc9b-9cd5-4f98-b693-607d925c4be8" (UID: "1958cc9b-9cd5-4f98-b693-607d925c4be8"). InnerVolumeSpecName "kube-api-access-b7v7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:00:56.456231 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.456149    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-token" (OuterVolumeSpecName: "istio-token") pod "1958cc9b-9cd5-4f98-b693-607d925c4be8" (UID: "1958cc9b-9cd5-4f98-b693-607d925c4be8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:00:56.456521 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.456496    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1958cc9b-9cd5-4f98-b693-607d925c4be8" (UID: "1958cc9b-9cd5-4f98-b693-607d925c4be8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:56.456613 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.456500    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1958cc9b-9cd5-4f98-b693-607d925c4be8" (UID: "1958cc9b-9cd5-4f98-b693-607d925c4be8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 11:00:56.497738 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.497711    2570 generic.go:356] "Generic (PLEG): container finished" podID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerID="a1502bb8307bb942cc3b8fb6b374413f28d5b8f07682eb6746393512c0d09121" exitCode=0
Sep 09 11:00:56.497850 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.497775    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"
Sep 09 11:00:56.497850 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.497784    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" event={"ID":"1958cc9b-9cd5-4f98-b693-607d925c4be8","Type":"ContainerDied","Data":"a1502bb8307bb942cc3b8fb6b374413f28d5b8f07682eb6746393512c0d09121"}
Sep 09 11:00:56.497850 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.497828    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd" event={"ID":"1958cc9b-9cd5-4f98-b693-607d925c4be8","Type":"ContainerDied","Data":"28935687756d6c3f066f1fdf9226781f8c7ac18d13291c48879a476025a1b36a"}
Sep 09 11:00:56.497850 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.497851    2570 scope.go:117] "RemoveContainer" containerID="a1502bb8307bb942cc3b8fb6b374413f28d5b8f07682eb6746393512c0d09121"
Sep 09 11:00:56.506695 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.506678    2570 scope.go:117] "RemoveContainer" containerID="51c2aa0f949ef25d8b17121ff6ad21599f577a266b9f57daa6dde269ff9fdc62"
Sep 09 11:00:56.514076 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.514060    2570 scope.go:117] "RemoveContainer" containerID="a1502bb8307bb942cc3b8fb6b374413f28d5b8f07682eb6746393512c0d09121"
Sep 09 11:00:56.514340 ip-10-0-128-193 kubenswrapper[2570]: E0909 11:00:56.514322    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1502bb8307bb942cc3b8fb6b374413f28d5b8f07682eb6746393512c0d09121\": container with ID starting with a1502bb8307bb942cc3b8fb6b374413f28d5b8f07682eb6746393512c0d09121 not found: ID does not exist" containerID="a1502bb8307bb942cc3b8fb6b374413f28d5b8f07682eb6746393512c0d09121"
Sep 09 11:00:56.514402 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.514348    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1502bb8307bb942cc3b8fb6b374413f28d5b8f07682eb6746393512c0d09121"} err="failed to get container status \"a1502bb8307bb942cc3b8fb6b374413f28d5b8f07682eb6746393512c0d09121\": rpc error: code = NotFound desc = could not find container \"a1502bb8307bb942cc3b8fb6b374413f28d5b8f07682eb6746393512c0d09121\": container with ID starting with a1502bb8307bb942cc3b8fb6b374413f28d5b8f07682eb6746393512c0d09121 not found: ID does not exist"
Sep 09 11:00:56.514402 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.514364    2570 scope.go:117] "RemoveContainer" containerID="51c2aa0f949ef25d8b17121ff6ad21599f577a266b9f57daa6dde269ff9fdc62"
Sep 09 11:00:56.514596 ip-10-0-128-193 kubenswrapper[2570]: E0909 11:00:56.514578    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c2aa0f949ef25d8b17121ff6ad21599f577a266b9f57daa6dde269ff9fdc62\": container with ID starting with 51c2aa0f949ef25d8b17121ff6ad21599f577a266b9f57daa6dde269ff9fdc62 not found: ID does not exist" containerID="51c2aa0f949ef25d8b17121ff6ad21599f577a266b9f57daa6dde269ff9fdc62"
Sep 09 11:00:56.514637 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.514601    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c2aa0f949ef25d8b17121ff6ad21599f577a266b9f57daa6dde269ff9fdc62"} err="failed to get container status \"51c2aa0f949ef25d8b17121ff6ad21599f577a266b9f57daa6dde269ff9fdc62\": rpc error: code = NotFound desc = could not find container \"51c2aa0f949ef25d8b17121ff6ad21599f577a266b9f57daa6dde269ff9fdc62\": container with ID starting with 51c2aa0f949ef25d8b17121ff6ad21599f577a266b9f57daa6dde269ff9fdc62 not found: ID does not exist"
Sep 09 11:00:56.520034 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.520013    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"]
Sep 09 11:00:56.527110 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.527090    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1eb21d-openshift-default-6df59d7f4b-t98xd"]
Sep 09 11:00:56.555560 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.555537    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-podinfo\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:56.555560 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.555558    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-envoy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:56.555673 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.555567    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7v7w\" (UniqueName: \"kubernetes.io/projected/1958cc9b-9cd5-4f98-b693-607d925c4be8-kube-api-access-b7v7w\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:56.555673 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.555578    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1958cc9b-9cd5-4f98-b693-607d925c4be8-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:00:56.839233 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:00:56.839206    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" path="/var/lib/kubelet/pods/1958cc9b-9cd5-4f98-b693-607d925c4be8/volumes"
Sep 09 11:01:01.152316 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:01.152289    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 11:01:01.153816 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:01.153794    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 11:01:01.158628 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:01.158611    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 11:01:01.159840 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:01.159823    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 11:01:01.960665 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:01.960616    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"]
Sep 09 11:01:01.960960 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:01.960921    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d" podUID="14e53582-dca0-4305-a883-a2d2e23eaa8f" containerName="maas-api" containerID="cri-o://a357004904d9da55e1e99c2b0fd6141fc2204f377808ccee7ba110c96e6dae4c" gracePeriod=30
Sep 09 11:01:02.217757 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.217687    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 11:01:02.306976 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.306945    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14e53582-dca0-4305-a883-a2d2e23eaa8f-metrics-certs\") pod \"14e53582-dca0-4305-a883-a2d2e23eaa8f\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") "
Sep 09 11:01:02.307117 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.307010    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn5lj\" (UniqueName: \"kubernetes.io/projected/14e53582-dca0-4305-a883-a2d2e23eaa8f-kube-api-access-qn5lj\") pod \"14e53582-dca0-4305-a883-a2d2e23eaa8f\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") "
Sep 09 11:01:02.307117 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.307037    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e53582-dca0-4305-a883-a2d2e23eaa8f-openshift-service-ca\") pod \"14e53582-dca0-4305-a883-a2d2e23eaa8f\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") "
Sep 09 11:01:02.307117 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.307096    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e53582-dca0-4305-a883-a2d2e23eaa8f-maas-api-tls\") pod \"14e53582-dca0-4305-a883-a2d2e23eaa8f\" (UID: \"14e53582-dca0-4305-a883-a2d2e23eaa8f\") "
Sep 09 11:01:02.307444 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.307420    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e53582-dca0-4305-a883-a2d2e23eaa8f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "14e53582-dca0-4305-a883-a2d2e23eaa8f" (UID: "14e53582-dca0-4305-a883-a2d2e23eaa8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 11:01:02.309280 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.309246    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e53582-dca0-4305-a883-a2d2e23eaa8f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "14e53582-dca0-4305-a883-a2d2e23eaa8f" (UID: "14e53582-dca0-4305-a883-a2d2e23eaa8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:01:02.309369 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.309284    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e53582-dca0-4305-a883-a2d2e23eaa8f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "14e53582-dca0-4305-a883-a2d2e23eaa8f" (UID: "14e53582-dca0-4305-a883-a2d2e23eaa8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:01:02.309419 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.309363    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e53582-dca0-4305-a883-a2d2e23eaa8f-kube-api-access-qn5lj" (OuterVolumeSpecName: "kube-api-access-qn5lj") pod "14e53582-dca0-4305-a883-a2d2e23eaa8f" (UID: "14e53582-dca0-4305-a883-a2d2e23eaa8f"). InnerVolumeSpecName "kube-api-access-qn5lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:01:02.408334 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.408295    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn5lj\" (UniqueName: \"kubernetes.io/projected/14e53582-dca0-4305-a883-a2d2e23eaa8f-kube-api-access-qn5lj\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:02.408334 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.408333    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e53582-dca0-4305-a883-a2d2e23eaa8f-openshift-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:02.408525 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.408349    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e53582-dca0-4305-a883-a2d2e23eaa8f-maas-api-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:02.408525 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.408361    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14e53582-dca0-4305-a883-a2d2e23eaa8f-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:02.527317 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.527280    2570 generic.go:356] "Generic (PLEG): container finished" podID="14e53582-dca0-4305-a883-a2d2e23eaa8f" containerID="a357004904d9da55e1e99c2b0fd6141fc2204f377808ccee7ba110c96e6dae4c" exitCode=0
Sep 09 11:01:02.527513 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.527321    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"
Sep 09 11:01:02.527513 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.527340    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d" event={"ID":"14e53582-dca0-4305-a883-a2d2e23eaa8f","Type":"ContainerDied","Data":"a357004904d9da55e1e99c2b0fd6141fc2204f377808ccee7ba110c96e6dae4c"}
Sep 09 11:01:02.527513 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.527416    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d" event={"ID":"14e53582-dca0-4305-a883-a2d2e23eaa8f","Type":"ContainerDied","Data":"0d47e0300d66d152abf5c3cc86dae3adb15f2bee24eb9bcb5329e61e80a29b6e"}
Sep 09 11:01:02.527513 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.527447    2570 scope.go:117] "RemoveContainer" containerID="a357004904d9da55e1e99c2b0fd6141fc2204f377808ccee7ba110c96e6dae4c"
Sep 09 11:01:02.537153 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.537133    2570 scope.go:117] "RemoveContainer" containerID="a357004904d9da55e1e99c2b0fd6141fc2204f377808ccee7ba110c96e6dae4c"
Sep 09 11:01:02.537437 ip-10-0-128-193 kubenswrapper[2570]: E0909 11:01:02.537418    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a357004904d9da55e1e99c2b0fd6141fc2204f377808ccee7ba110c96e6dae4c\": container with ID starting with a357004904d9da55e1e99c2b0fd6141fc2204f377808ccee7ba110c96e6dae4c not found: ID does not exist" containerID="a357004904d9da55e1e99c2b0fd6141fc2204f377808ccee7ba110c96e6dae4c"
Sep 09 11:01:02.537492 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.537445    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a357004904d9da55e1e99c2b0fd6141fc2204f377808ccee7ba110c96e6dae4c"} err="failed to get container status \"a357004904d9da55e1e99c2b0fd6141fc2204f377808ccee7ba110c96e6dae4c\": rpc error: code = NotFound desc = could not find container \"a357004904d9da55e1e99c2b0fd6141fc2204f377808ccee7ba110c96e6dae4c\": container with ID starting with a357004904d9da55e1e99c2b0fd6141fc2204f377808ccee7ba110c96e6dae4c not found: ID does not exist"
Sep 09 11:01:02.551113 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.551088    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"]
Sep 09 11:01:02.555648 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.555626    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-535b75-67c9757947-tb48d"]
Sep 09 11:01:02.727162 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.727128    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:01:02.798139 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.798055    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:01:02.838947 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:02.838912    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14e53582-dca0-4305-a883-a2d2e23eaa8f" path="/var/lib/kubelet/pods/14e53582-dca0-4305-a883-a2d2e23eaa8f/volumes"
Sep 09 11:01:09.666507 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:09.666478    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"]
Sep 09 11:01:09.667084 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:09.666684    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy" containerID="cri-o://bd2a08a8bb1bdb18381a2483ba141bc90063962ea3c5c9f6e2fd74a3b4c516c7" gracePeriod=30
Sep 09 11:01:09.700716 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:09.700685    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:01:09.780373 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:09.780339    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:01:12.441718 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.441689    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-ln86k"]
Sep 09 11:01:12.442113 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.441899    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68dd87c4cd-ln86k" podUID="c52d5c22-b63c-4998-970e-70d31838fd5e" containerName="manager" containerID="cri-o://4455b0356978e769ec98de5ac0cce98c394ec7fed7a5f92dbadbd1ddb8226b5e" gracePeriod=30
Sep 09 11:01:12.577526 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.577494    2570 generic.go:356] "Generic (PLEG): container finished" podID="c52d5c22-b63c-4998-970e-70d31838fd5e" containerID="4455b0356978e769ec98de5ac0cce98c394ec7fed7a5f92dbadbd1ddb8226b5e" exitCode=0
Sep 09 11:01:12.577698 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.577570    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68dd87c4cd-ln86k" event={"ID":"c52d5c22-b63c-4998-970e-70d31838fd5e","Type":"ContainerDied","Data":"4455b0356978e769ec98de5ac0cce98c394ec7fed7a5f92dbadbd1ddb8226b5e"}
Sep 09 11:01:12.694293 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.694236    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 11:01:12.696408 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.696390    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c52d5c22-b63c-4998-970e-70d31838fd5e-metrics-certs\") pod \"c52d5c22-b63c-4998-970e-70d31838fd5e\" (UID: \"c52d5c22-b63c-4998-970e-70d31838fd5e\") "
Sep 09 11:01:12.696462 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.696449    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c52d5c22-b63c-4998-970e-70d31838fd5e-webhook-cert\") pod \"c52d5c22-b63c-4998-970e-70d31838fd5e\" (UID: \"c52d5c22-b63c-4998-970e-70d31838fd5e\") "
Sep 09 11:01:12.696502 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.696483    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72xx2\" (UniqueName: \"kubernetes.io/projected/c52d5c22-b63c-4998-970e-70d31838fd5e-kube-api-access-72xx2\") pod \"c52d5c22-b63c-4998-970e-70d31838fd5e\" (UID: \"c52d5c22-b63c-4998-970e-70d31838fd5e\") "
Sep 09 11:01:12.698577 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.698550    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52d5c22-b63c-4998-970e-70d31838fd5e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c52d5c22-b63c-4998-970e-70d31838fd5e" (UID: "c52d5c22-b63c-4998-970e-70d31838fd5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:01:12.698692 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.698647    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52d5c22-b63c-4998-970e-70d31838fd5e-kube-api-access-72xx2" (OuterVolumeSpecName: "kube-api-access-72xx2") pod "c52d5c22-b63c-4998-970e-70d31838fd5e" (UID: "c52d5c22-b63c-4998-970e-70d31838fd5e"). InnerVolumeSpecName "kube-api-access-72xx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:01:12.698692 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.698674    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52d5c22-b63c-4998-970e-70d31838fd5e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c52d5c22-b63c-4998-970e-70d31838fd5e" (UID: "c52d5c22-b63c-4998-970e-70d31838fd5e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:01:12.797277 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.797240    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c52d5c22-b63c-4998-970e-70d31838fd5e-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:12.797277 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.797268    2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c52d5c22-b63c-4998-970e-70d31838fd5e-webhook-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:12.797277 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:12.797277    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72xx2\" (UniqueName: \"kubernetes.io/projected/c52d5c22-b63c-4998-970e-70d31838fd5e-kube-api-access-72xx2\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:13.584012 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:13.583983    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68dd87c4cd-ln86k"
Sep 09 11:01:13.584012 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:13.584027    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68dd87c4cd-ln86k" event={"ID":"c52d5c22-b63c-4998-970e-70d31838fd5e","Type":"ContainerDied","Data":"8954ab842308007e3b7c123dcc8c97ad81b8603b87130995b367bcd3cd0abf67"}
Sep 09 11:01:13.584607 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:13.584093    2570 scope.go:117] "RemoveContainer" containerID="4455b0356978e769ec98de5ac0cce98c394ec7fed7a5f92dbadbd1ddb8226b5e"
Sep 09 11:01:13.601507 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:13.601484    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-ln86k"]
Sep 09 11:01:13.605369 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:13.605345    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-ln86k"]
Sep 09 11:01:14.631870 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:14.631839    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 11:01:14.632234 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:14.631891    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 11:01:14.841191 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:14.841163    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c52d5c22-b63c-4998-970e-70d31838fd5e" path="/var/lib/kubelet/pods/c52d5c22-b63c-4998-970e-70d31838fd5e/volumes"
Sep 09 11:01:14.915248 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:14.915229    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 11:01:15.014263 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.014238    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-workload-certs\") pod \"c391eaca-317f-4de8-994b-f6d246ae8f9b\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") "
Sep 09 11:01:15.014471 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.014274    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-token\") pod \"c391eaca-317f-4de8-994b-f6d246ae8f9b\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") "
Sep 09 11:01:15.014471 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.014319    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-envoy\") pod \"c391eaca-317f-4de8-994b-f6d246ae8f9b\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") "
Sep 09 11:01:15.014471 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.014359    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-data\") pod \"c391eaca-317f-4de8-994b-f6d246ae8f9b\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") "
Sep 09 11:01:15.014471 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.014391    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-574dt\" (UniqueName: \"kubernetes.io/projected/c391eaca-317f-4de8-994b-f6d246ae8f9b-kube-api-access-574dt\") pod \"c391eaca-317f-4de8-994b-f6d246ae8f9b\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") "
Sep 09 11:01:15.014471 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.014422    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-credential-socket\") pod \"c391eaca-317f-4de8-994b-f6d246ae8f9b\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") "
Sep 09 11:01:15.014471 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.014450    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-podinfo\") pod \"c391eaca-317f-4de8-994b-f6d246ae8f9b\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") "
Sep 09 11:01:15.014835 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.014524    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-workload-socket\") pod \"c391eaca-317f-4de8-994b-f6d246ae8f9b\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") "
Sep 09 11:01:15.014835 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.014520    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c391eaca-317f-4de8-994b-f6d246ae8f9b" (UID: "c391eaca-317f-4de8-994b-f6d246ae8f9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:01:15.014835 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.014615    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c391eaca-317f-4de8-994b-f6d246ae8f9b-istiod-ca-cert\") pod \"c391eaca-317f-4de8-994b-f6d246ae8f9b\" (UID: \"c391eaca-317f-4de8-994b-f6d246ae8f9b\") "
Sep 09 11:01:15.014835 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.014658    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-data" (OuterVolumeSpecName: "istio-data") pod "c391eaca-317f-4de8-994b-f6d246ae8f9b" (UID: "c391eaca-317f-4de8-994b-f6d246ae8f9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:01:15.014835 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.014721    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c391eaca-317f-4de8-994b-f6d246ae8f9b" (UID: "c391eaca-317f-4de8-994b-f6d246ae8f9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:01:15.014835 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.014747    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c391eaca-317f-4de8-994b-f6d246ae8f9b" (UID: "c391eaca-317f-4de8-994b-f6d246ae8f9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:01:15.015130 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.015022    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-data\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:15.015130 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.015041    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-credential-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:15.015130 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.015044    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c391eaca-317f-4de8-994b-f6d246ae8f9b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c391eaca-317f-4de8-994b-f6d246ae8f9b" (UID: "c391eaca-317f-4de8-994b-f6d246ae8f9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 11:01:15.015130 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.015056    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-workload-socket\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:15.015130 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.015070    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-workload-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:15.016563 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.016537    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c391eaca-317f-4de8-994b-f6d246ae8f9b" (UID: "c391eaca-317f-4de8-994b-f6d246ae8f9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:01:15.016668 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.016537    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c391eaca-317f-4de8-994b-f6d246ae8f9b" (UID: "c391eaca-317f-4de8-994b-f6d246ae8f9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 11:01:15.016905 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.016881    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-token" (OuterVolumeSpecName: "istio-token") pod "c391eaca-317f-4de8-994b-f6d246ae8f9b" (UID: "c391eaca-317f-4de8-994b-f6d246ae8f9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:01:15.016905 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.016894    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c391eaca-317f-4de8-994b-f6d246ae8f9b-kube-api-access-574dt" (OuterVolumeSpecName: "kube-api-access-574dt") pod "c391eaca-317f-4de8-994b-f6d246ae8f9b" (UID: "c391eaca-317f-4de8-994b-f6d246ae8f9b"). InnerVolumeSpecName "kube-api-access-574dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:01:15.116230 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.116174    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-574dt\" (UniqueName: \"kubernetes.io/projected/c391eaca-317f-4de8-994b-f6d246ae8f9b-kube-api-access-574dt\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:15.116230 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.116231    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-podinfo\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:15.116440 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.116246    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c391eaca-317f-4de8-994b-f6d246ae8f9b-istiod-ca-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:15.116440 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.116260    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-token\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:15.116440 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.116274    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c391eaca-317f-4de8-994b-f6d246ae8f9b-istio-envoy\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:01:15.596742 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.596706    2570 generic.go:356] "Generic (PLEG): container finished" podID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerID="bd2a08a8bb1bdb18381a2483ba141bc90063962ea3c5c9f6e2fd74a3b4c516c7" exitCode=0
Sep 09 11:01:15.596939 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.596788    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"
Sep 09 11:01:15.596939 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.596812    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" event={"ID":"c391eaca-317f-4de8-994b-f6d246ae8f9b","Type":"ContainerDied","Data":"bd2a08a8bb1bdb18381a2483ba141bc90063962ea3c5c9f6e2fd74a3b4c516c7"}
Sep 09 11:01:15.596939 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.596885    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l" event={"ID":"c391eaca-317f-4de8-994b-f6d246ae8f9b","Type":"ContainerDied","Data":"c1670a6ea6276648a3abf5094f2a667e9ef3a0212ac41e1793e5dc464034c977"}
Sep 09 11:01:15.596939 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.596922    2570 scope.go:117] "RemoveContainer" containerID="bd2a08a8bb1bdb18381a2483ba141bc90063962ea3c5c9f6e2fd74a3b4c516c7"
Sep 09 11:01:15.607841 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.607783    2570 scope.go:117] "RemoveContainer" containerID="95b966dc1777b7745de817054f8ce1f16881b2cfefdd281ede0062be3ef3b844"
Sep 09 11:01:15.615302 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.615284    2570 scope.go:117] "RemoveContainer" containerID="bd2a08a8bb1bdb18381a2483ba141bc90063962ea3c5c9f6e2fd74a3b4c516c7"
Sep 09 11:01:15.615565 ip-10-0-128-193 kubenswrapper[2570]: E0909 11:01:15.615548    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2a08a8bb1bdb18381a2483ba141bc90063962ea3c5c9f6e2fd74a3b4c516c7\": container with ID starting with bd2a08a8bb1bdb18381a2483ba141bc90063962ea3c5c9f6e2fd74a3b4c516c7 not found: ID does not exist" containerID="bd2a08a8bb1bdb18381a2483ba141bc90063962ea3c5c9f6e2fd74a3b4c516c7"
Sep 09 11:01:15.615607 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.615573    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2a08a8bb1bdb18381a2483ba141bc90063962ea3c5c9f6e2fd74a3b4c516c7"} err="failed to get container status \"bd2a08a8bb1bdb18381a2483ba141bc90063962ea3c5c9f6e2fd74a3b4c516c7\": rpc error: code = NotFound desc = could not find container \"bd2a08a8bb1bdb18381a2483ba141bc90063962ea3c5c9f6e2fd74a3b4c516c7\": container with ID starting with bd2a08a8bb1bdb18381a2483ba141bc90063962ea3c5c9f6e2fd74a3b4c516c7 not found: ID does not exist"
Sep 09 11:01:15.615607 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.615592    2570 scope.go:117] "RemoveContainer" containerID="95b966dc1777b7745de817054f8ce1f16881b2cfefdd281ede0062be3ef3b844"
Sep 09 11:01:15.615866 ip-10-0-128-193 kubenswrapper[2570]: E0909 11:01:15.615849    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b966dc1777b7745de817054f8ce1f16881b2cfefdd281ede0062be3ef3b844\": container with ID starting with 95b966dc1777b7745de817054f8ce1f16881b2cfefdd281ede0062be3ef3b844 not found: ID does not exist" containerID="95b966dc1777b7745de817054f8ce1f16881b2cfefdd281ede0062be3ef3b844"
Sep 09 11:01:15.615940 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.615870    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b966dc1777b7745de817054f8ce1f16881b2cfefdd281ede0062be3ef3b844"} err="failed to get container status \"95b966dc1777b7745de817054f8ce1f16881b2cfefdd281ede0062be3ef3b844\": rpc error: code = NotFound desc = could not find container \"95b966dc1777b7745de817054f8ce1f16881b2cfefdd281ede0062be3ef3b844\": container with ID starting with 95b966dc1777b7745de817054f8ce1f16881b2cfefdd281ede0062be3ef3b844 not found: ID does not exist"
Sep 09 11:01:15.622885 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.622862    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"]
Sep 09 11:01:15.629245 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:15.629225    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-535b75-openshift-default-778d669cf6-jtk4l"]
Sep 09 11:01:16.838901 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:01:16.838869    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" path="/var/lib/kubelet/pods/c391eaca-317f-4de8-994b-f6d246ae8f9b/volumes"
Sep 09 11:02:14.563902 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.563871    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-5cqwl"]
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564205    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564216    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564225    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564231    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564239    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564246    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564255    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bc39842-9218-4d3c-8123-0e4788203dc1" containerName="payload-pre-processing"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564261    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bc39842-9218-4d3c-8123-0e4788203dc1" containerName="payload-pre-processing"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564266    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564271    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564278    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerName="istio-proxy"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564283    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerName="istio-proxy"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564290    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52d5c22-b63c-4998-970e-70d31838fd5e" containerName="manager"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564296    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52d5c22-b63c-4998-970e-70d31838fd5e" containerName="manager"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564302    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e53582-dca0-4305-a883-a2d2e23eaa8f" containerName="maas-api"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564307    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e53582-dca0-4305-a883-a2d2e23eaa8f" containerName="maas-api"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564317    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd59a316-08c5-4d29-9eb6-6400a834c8dc" containerName="maas-api"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564322    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd59a316-08c5-4d29-9eb6-6400a834c8dc" containerName="maas-api"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564331    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerName="istio-proxy"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564336    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerName="istio-proxy"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564343    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerName="istio-proxy"
Sep 09 11:02:14.564379 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564349    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerName="istio-proxy"
Sep 09 11:02:14.565246 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564413    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerName="istio-proxy"
Sep 09 11:02:14.565246 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564422    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy"
Sep 09 11:02:14.565246 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564427    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerName="istio-proxy"
Sep 09 11:02:14.565246 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564433    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="14e53582-dca0-4305-a883-a2d2e23eaa8f" containerName="maas-api"
Sep 09 11:02:14.565246 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564439    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfa1f0c-b258-439b-b9b0-4341bdb09aa4" containerName="istio-proxy"
Sep 09 11:02:14.565246 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564444    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bc39842-9218-4d3c-8123-0e4788203dc1" containerName="payload-pre-processing"
Sep 09 11:02:14.565246 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564449    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd59a316-08c5-4d29-9eb6-6400a834c8dc" containerName="maas-api"
Sep 09 11:02:14.565246 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564454    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerName="istio-proxy"
Sep 09 11:02:14.565246 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564460    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy"
Sep 09 11:02:14.565246 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564466    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1958cc9b-9cd5-4f98-b693-607d925c4be8" containerName="istio-proxy"
Sep 09 11:02:14.565246 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564471    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52d5c22-b63c-4998-970e-70d31838fd5e" containerName="manager"
Sep 09 11:02:14.565246 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.564476    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c391eaca-317f-4de8-994b-f6d246ae8f9b" containerName="istio-proxy"
Sep 09 11:02:14.566721 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.566705    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:14.570749 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.570724    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lr9tv\""
Sep 09 11:02:14.570909 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.570845    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 11:02:14.570909 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.570889    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 11:02:14.571025 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.570966    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 11:02:14.576155 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.576135    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-5cqwl"]
Sep 09 11:02:14.622318 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.622288    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67rg\" (UniqueName: \"kubernetes.io/projected/6da7f581-3170-4d92-943e-ffe573acefd7-kube-api-access-x67rg\") pod \"maas-controller-68dd87c4cd-5cqwl\" (UID: \"6da7f581-3170-4d92-943e-ffe573acefd7\") " pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:14.622459 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.622356    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6da7f581-3170-4d92-943e-ffe573acefd7-webhook-cert\") pod \"maas-controller-68dd87c4cd-5cqwl\" (UID: \"6da7f581-3170-4d92-943e-ffe573acefd7\") " pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:14.622459 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.622414    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da7f581-3170-4d92-943e-ffe573acefd7-metrics-certs\") pod \"maas-controller-68dd87c4cd-5cqwl\" (UID: \"6da7f581-3170-4d92-943e-ffe573acefd7\") " pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:14.723613 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.723590    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67rg\" (UniqueName: \"kubernetes.io/projected/6da7f581-3170-4d92-943e-ffe573acefd7-kube-api-access-x67rg\") pod \"maas-controller-68dd87c4cd-5cqwl\" (UID: \"6da7f581-3170-4d92-943e-ffe573acefd7\") " pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:14.723728 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.723638    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6da7f581-3170-4d92-943e-ffe573acefd7-webhook-cert\") pod \"maas-controller-68dd87c4cd-5cqwl\" (UID: \"6da7f581-3170-4d92-943e-ffe573acefd7\") " pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:14.723728 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.723672    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da7f581-3170-4d92-943e-ffe573acefd7-metrics-certs\") pod \"maas-controller-68dd87c4cd-5cqwl\" (UID: \"6da7f581-3170-4d92-943e-ffe573acefd7\") " pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:14.726121 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.726101    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da7f581-3170-4d92-943e-ffe573acefd7-metrics-certs\") pod \"maas-controller-68dd87c4cd-5cqwl\" (UID: \"6da7f581-3170-4d92-943e-ffe573acefd7\") " pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:14.726222 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.726200    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6da7f581-3170-4d92-943e-ffe573acefd7-webhook-cert\") pod \"maas-controller-68dd87c4cd-5cqwl\" (UID: \"6da7f581-3170-4d92-943e-ffe573acefd7\") " pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:14.733337 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.733319    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67rg\" (UniqueName: \"kubernetes.io/projected/6da7f581-3170-4d92-943e-ffe573acefd7-kube-api-access-x67rg\") pod \"maas-controller-68dd87c4cd-5cqwl\" (UID: \"6da7f581-3170-4d92-943e-ffe573acefd7\") " pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:14.878149 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:14.878085    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:15.147462 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:15.147439    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-5cqwl"]
Sep 09 11:02:15.149081 ip-10-0-128-193 kubenswrapper[2570]: W0909 11:02:15.149049    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da7f581_3170_4d92_943e_ffe573acefd7.slice/crio-f90dddb360e949a7e2815a4c8c1bd5418bfdfa21636805e6d0029de3adfec77b WatchSource:0}: Error finding container f90dddb360e949a7e2815a4c8c1bd5418bfdfa21636805e6d0029de3adfec77b: Status 404 returned error can't find the container with id f90dddb360e949a7e2815a4c8c1bd5418bfdfa21636805e6d0029de3adfec77b
Sep 09 11:02:15.150369 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:15.150355    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 11:02:15.875979 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:15.875936    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68dd87c4cd-5cqwl" event={"ID":"6da7f581-3170-4d92-943e-ffe573acefd7","Type":"ContainerStarted","Data":"f90dddb360e949a7e2815a4c8c1bd5418bfdfa21636805e6d0029de3adfec77b"}
Sep 09 11:02:18.890239 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:18.890202    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68dd87c4cd-5cqwl" event={"ID":"6da7f581-3170-4d92-943e-ffe573acefd7","Type":"ContainerStarted","Data":"d5074382ec9f1b0a85346c1c1ad8253a552a07a98ea196bc07f5eb1686a05604"}
Sep 09 11:02:18.890605 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:18.890274    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:18.911677 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:18.911629    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68dd87c4cd-5cqwl" podStartSLOduration=1.6864867540000001 podStartE2EDuration="4.911614236s" totalImagesPullingTime="3.225127482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 11:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:02:18.909087408 +0000 UTC m=+2178.572726393" watchObservedRunningTime="2026-09-09 11:02:18.911614236 +0000 UTC m=+2178.575253210"
Sep 09 11:02:29.897755 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:29.897722    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:38.852436 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:38.852398    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:02:39.357447 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.357410    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-5cqwl"]
Sep 09 11:02:39.357633 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.357609    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68dd87c4cd-5cqwl" podUID="6da7f581-3170-4d92-943e-ffe573acefd7" containerName="manager" containerID="cri-o://d5074382ec9f1b0a85346c1c1ad8253a552a07a98ea196bc07f5eb1686a05604" gracePeriod=30
Sep 09 11:02:39.599910 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.599887    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:39.728891 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.728807    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da7f581-3170-4d92-943e-ffe573acefd7-metrics-certs\") pod \"6da7f581-3170-4d92-943e-ffe573acefd7\" (UID: \"6da7f581-3170-4d92-943e-ffe573acefd7\") "
Sep 09 11:02:39.729066 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.728898    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x67rg\" (UniqueName: \"kubernetes.io/projected/6da7f581-3170-4d92-943e-ffe573acefd7-kube-api-access-x67rg\") pod \"6da7f581-3170-4d92-943e-ffe573acefd7\" (UID: \"6da7f581-3170-4d92-943e-ffe573acefd7\") "
Sep 09 11:02:39.729066 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.728924    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6da7f581-3170-4d92-943e-ffe573acefd7-webhook-cert\") pod \"6da7f581-3170-4d92-943e-ffe573acefd7\" (UID: \"6da7f581-3170-4d92-943e-ffe573acefd7\") "
Sep 09 11:02:39.731019 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.730979    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da7f581-3170-4d92-943e-ffe573acefd7-kube-api-access-x67rg" (OuterVolumeSpecName: "kube-api-access-x67rg") pod "6da7f581-3170-4d92-943e-ffe573acefd7" (UID: "6da7f581-3170-4d92-943e-ffe573acefd7"). InnerVolumeSpecName "kube-api-access-x67rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:02:39.731135 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.731021    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da7f581-3170-4d92-943e-ffe573acefd7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6da7f581-3170-4d92-943e-ffe573acefd7" (UID: "6da7f581-3170-4d92-943e-ffe573acefd7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:02:39.731135 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.731036    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da7f581-3170-4d92-943e-ffe573acefd7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6da7f581-3170-4d92-943e-ffe573acefd7" (UID: "6da7f581-3170-4d92-943e-ffe573acefd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:02:39.830154 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.830120    2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6da7f581-3170-4d92-943e-ffe573acefd7-webhook-cert\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:02:39.830154 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.830147    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da7f581-3170-4d92-943e-ffe573acefd7-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:02:39.830154 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.830159    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x67rg\" (UniqueName: \"kubernetes.io/projected/6da7f581-3170-4d92-943e-ffe573acefd7-kube-api-access-x67rg\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:02:39.985420 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.985333    2570 generic.go:356] "Generic (PLEG): container finished" podID="6da7f581-3170-4d92-943e-ffe573acefd7" containerID="d5074382ec9f1b0a85346c1c1ad8253a552a07a98ea196bc07f5eb1686a05604" exitCode=0
Sep 09 11:02:39.985420 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.985378    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68dd87c4cd-5cqwl"
Sep 09 11:02:39.985912 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.985416    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68dd87c4cd-5cqwl" event={"ID":"6da7f581-3170-4d92-943e-ffe573acefd7","Type":"ContainerDied","Data":"d5074382ec9f1b0a85346c1c1ad8253a552a07a98ea196bc07f5eb1686a05604"}
Sep 09 11:02:39.985912 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.985466    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68dd87c4cd-5cqwl" event={"ID":"6da7f581-3170-4d92-943e-ffe573acefd7","Type":"ContainerDied","Data":"f90dddb360e949a7e2815a4c8c1bd5418bfdfa21636805e6d0029de3adfec77b"}
Sep 09 11:02:39.985912 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.985489    2570 scope.go:117] "RemoveContainer" containerID="d5074382ec9f1b0a85346c1c1ad8253a552a07a98ea196bc07f5eb1686a05604"
Sep 09 11:02:39.994960 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.994944    2570 scope.go:117] "RemoveContainer" containerID="d5074382ec9f1b0a85346c1c1ad8253a552a07a98ea196bc07f5eb1686a05604"
Sep 09 11:02:39.995231 ip-10-0-128-193 kubenswrapper[2570]: E0909 11:02:39.995214    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5074382ec9f1b0a85346c1c1ad8253a552a07a98ea196bc07f5eb1686a05604\": container with ID starting with d5074382ec9f1b0a85346c1c1ad8253a552a07a98ea196bc07f5eb1686a05604 not found: ID does not exist" containerID="d5074382ec9f1b0a85346c1c1ad8253a552a07a98ea196bc07f5eb1686a05604"
Sep 09 11:02:39.995294 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:39.995244    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5074382ec9f1b0a85346c1c1ad8253a552a07a98ea196bc07f5eb1686a05604"} err="failed to get container status \"d5074382ec9f1b0a85346c1c1ad8253a552a07a98ea196bc07f5eb1686a05604\": rpc error: code = NotFound desc = could not find container \"d5074382ec9f1b0a85346c1c1ad8253a552a07a98ea196bc07f5eb1686a05604\": container with ID starting with d5074382ec9f1b0a85346c1c1ad8253a552a07a98ea196bc07f5eb1686a05604 not found: ID does not exist"
Sep 09 11:02:40.008019 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.007995    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-5cqwl"]
Sep 09 11:02:40.015604 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.015574    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-5cqwl"]
Sep 09 11:02:40.841039 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.841011    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6da7f581-3170-4d92-943e-ffe573acefd7" path="/var/lib/kubelet/pods/6da7f581-3170-4d92-943e-ffe573acefd7/volumes"
Sep 09 11:02:40.979878 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.979846    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-66t9l"]
Sep 09 11:02:40.980167 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.980155    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6da7f581-3170-4d92-943e-ffe573acefd7" containerName="manager"
Sep 09 11:02:40.980208 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.980168    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6da7f581-3170-4d92-943e-ffe573acefd7" containerName="manager"
Sep 09 11:02:40.980208 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.980196    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerName="istio-proxy"
Sep 09 11:02:40.980208 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.980201    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cdcc01d-76ba-4ac4-aa33-c03307d3caca" containerName="istio-proxy"
Sep 09 11:02:40.980306 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.980263    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6da7f581-3170-4d92-943e-ffe573acefd7" containerName="manager"
Sep 09 11:02:40.984179 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.984162    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68dd87c4cd-66t9l"
Sep 09 11:02:40.988436 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.988413    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 11:02:40.988823 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.988432    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 11:02:40.988823 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.988487    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 11:02:40.988823 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.988491    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lr9tv\""
Sep 09 11:02:40.994714 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:40.994690    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-66t9l"]
Sep 09 11:02:41.142500 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:41.142419    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/949317c2-d4da-4ef6-9ee4-68a565088b20-metrics-certs\") pod \"maas-controller-68dd87c4cd-66t9l\" (UID: \"949317c2-d4da-4ef6-9ee4-68a565088b20\") " pod="opendatahub/maas-controller-68dd87c4cd-66t9l"
Sep 09 11:02:41.142500 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:41.142478    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/949317c2-d4da-4ef6-9ee4-68a565088b20-webhook-cert\") pod \"maas-controller-68dd87c4cd-66t9l\" (UID: \"949317c2-d4da-4ef6-9ee4-68a565088b20\") " pod="opendatahub/maas-controller-68dd87c4cd-66t9l"
Sep 09 11:02:41.142689 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:41.142523    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l5zc\" (UniqueName: \"kubernetes.io/projected/949317c2-d4da-4ef6-9ee4-68a565088b20-kube-api-access-4l5zc\") pod \"maas-controller-68dd87c4cd-66t9l\" (UID: \"949317c2-d4da-4ef6-9ee4-68a565088b20\") " pod="opendatahub/maas-controller-68dd87c4cd-66t9l"
Sep 09 11:02:41.243328 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:41.243302    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/949317c2-d4da-4ef6-9ee4-68a565088b20-metrics-certs\") pod \"maas-controller-68dd87c4cd-66t9l\" (UID: \"949317c2-d4da-4ef6-9ee4-68a565088b20\") " pod="opendatahub/maas-controller-68dd87c4cd-66t9l"
Sep 09 11:02:41.243452 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:41.243349    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/949317c2-d4da-4ef6-9ee4-68a565088b20-webhook-cert\") pod \"maas-controller-68dd87c4cd-66t9l\" (UID: \"949317c2-d4da-4ef6-9ee4-68a565088b20\") " pod="opendatahub/maas-controller-68dd87c4cd-66t9l"
Sep 09 11:02:41.243452 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:41.243381    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l5zc\" (UniqueName: \"kubernetes.io/projected/949317c2-d4da-4ef6-9ee4-68a565088b20-kube-api-access-4l5zc\") pod \"maas-controller-68dd87c4cd-66t9l\" (UID: \"949317c2-d4da-4ef6-9ee4-68a565088b20\") " pod="opendatahub/maas-controller-68dd87c4cd-66t9l"
Sep 09 11:02:41.245706 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:41.245681    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/949317c2-d4da-4ef6-9ee4-68a565088b20-metrics-certs\") pod \"maas-controller-68dd87c4cd-66t9l\" (UID: \"949317c2-d4da-4ef6-9ee4-68a565088b20\") " pod="opendatahub/maas-controller-68dd87c4cd-66t9l"
Sep 09 11:02:41.245827 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:41.245713    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/949317c2-d4da-4ef6-9ee4-68a565088b20-webhook-cert\") pod \"maas-controller-68dd87c4cd-66t9l\" (UID: \"949317c2-d4da-4ef6-9ee4-68a565088b20\") " pod="opendatahub/maas-controller-68dd87c4cd-66t9l"
Sep 09 11:02:41.252052 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:41.252030    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l5zc\" (UniqueName: \"kubernetes.io/projected/949317c2-d4da-4ef6-9ee4-68a565088b20-kube-api-access-4l5zc\") pod \"maas-controller-68dd87c4cd-66t9l\" (UID: \"949317c2-d4da-4ef6-9ee4-68a565088b20\") " pod="opendatahub/maas-controller-68dd87c4cd-66t9l"
Sep 09 11:02:41.295851 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:41.295823    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68dd87c4cd-66t9l"
Sep 09 11:02:41.563910 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:41.563887    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68dd87c4cd-66t9l"]
Sep 09 11:02:41.565436 ip-10-0-128-193 kubenswrapper[2570]: W0909 11:02:41.565408    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949317c2_d4da_4ef6_9ee4_68a565088b20.slice/crio-fbcbad8edec5833d08ccf08071147e64ae48cd1a36af32e7856eb51fcf069ada WatchSource:0}: Error finding container fbcbad8edec5833d08ccf08071147e64ae48cd1a36af32e7856eb51fcf069ada: Status 404 returned error can't find the container with id fbcbad8edec5833d08ccf08071147e64ae48cd1a36af32e7856eb51fcf069ada
Sep 09 11:02:41.998457 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:41.998424    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68dd87c4cd-66t9l" event={"ID":"949317c2-d4da-4ef6-9ee4-68a565088b20","Type":"ContainerStarted","Data":"fbcbad8edec5833d08ccf08071147e64ae48cd1a36af32e7856eb51fcf069ada"}
Sep 09 11:02:42.623482 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:42.623452    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:02:44.008901 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:44.008867    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68dd87c4cd-66t9l" event={"ID":"949317c2-d4da-4ef6-9ee4-68a565088b20","Type":"ContainerStarted","Data":"d6af2a9f4d9c0b4e346f4882f471f21558d01ebad3b42b03add2028cd9f1d2e8"}
Sep 09 11:02:44.009255 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:44.008984    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68dd87c4cd-66t9l"
Sep 09 11:02:44.029733 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:44.029686    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68dd87c4cd-66t9l" podStartSLOduration=2.56978242 podStartE2EDuration="4.029672229s" totalImagesPullingTime="1.459889809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 11:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:02:44.027968133 +0000 UTC m=+2203.691607108" watchObservedRunningTime="2026-09-09 11:02:44.029672229 +0000 UTC m=+2203.693311203"
Sep 09 11:02:55.016741 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:55.016714    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68dd87c4cd-66t9l"
Sep 09 11:02:56.260547 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:02:56.260517    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:04:16.003679 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:04:16.003645    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:04:56.742858 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:04:56.742739    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:05:29.133093 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.133063    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"]
Sep 09 11:05:29.135867 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.135848    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.140931 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.140913    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fe318cb6-kserve-self-signed-certs\""
Sep 09 11:05:29.153798 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.153756    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"]
Sep 09 11:05:29.197356 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.197327    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:05:29.274757 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.274728    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-tmp-dir\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.274919 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.274789    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-model-cache\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.274919 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.274838    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-dshm\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.274919 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.274866    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbwr\" (UniqueName: \"kubernetes.io/projected/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-kube-api-access-wzbwr\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.274919 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.274906    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-home\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.275055 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.274924    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-tls-certs\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.375677 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.375648    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-model-cache\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.375888 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.375695    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-dshm\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.375888 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.375726    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbwr\" (UniqueName: \"kubernetes.io/projected/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-kube-api-access-wzbwr\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.375888 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.375803    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-home\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.375888 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.375831    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-tls-certs\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.376115 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.375940    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-tmp-dir\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.376115 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.376041    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-model-cache\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.376232 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.376189    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-home\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.376288 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.376272    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-tmp-dir\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.377920 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.377900    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-dshm\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.378319 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.378301    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-tls-certs\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.389450 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.389400    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbwr\" (UniqueName: \"kubernetes.io/projected/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-kube-api-access-wzbwr\") pod \"e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") " pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.445406 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.445387    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:29.738133 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.738107    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"]
Sep 09 11:05:29.739576 ip-10-0-128-193 kubenswrapper[2570]: W0909 11:05:29.739549    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb1c68b3_89fc_46f9_b147_f23fe06e1b73.slice/crio-399295a5066398c5774f8386526b67e59d4cd681ddf06527a00acd564af0be7f WatchSource:0}: Error finding container 399295a5066398c5774f8386526b67e59d4cd681ddf06527a00acd564af0be7f: Status 404 returned error can't find the container with id 399295a5066398c5774f8386526b67e59d4cd681ddf06527a00acd564af0be7f
Sep 09 11:05:29.786358 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:29.786325    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v" event={"ID":"fb1c68b3-89fc-46f9-b147-f23fe06e1b73","Type":"ContainerStarted","Data":"399295a5066398c5774f8386526b67e59d4cd681ddf06527a00acd564af0be7f"}
Sep 09 11:05:30.792888 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:30.792850    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v" event={"ID":"fb1c68b3-89fc-46f9-b147-f23fe06e1b73","Type":"ContainerStarted","Data":"79d6fbe5e754264bdf19f6a75894cdc6f881283038a8a8cacf790523fd7f22aa"}
Sep 09 11:05:30.841055 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:30.841006    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v" podStartSLOduration=1.8409923639999999 podStartE2EDuration="1.840992364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 11:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:05:30.839738298 +0000 UTC m=+2370.503377272" watchObservedRunningTime="2026-09-09 11:05:30.840992364 +0000 UTC m=+2370.504631335"
Sep 09 11:05:33.630088 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:33.630043    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:05:39.446187 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:39.446146    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:39.446563 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:39.446207    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:39.458942 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:39.458914    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:39.845308 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:39.845282    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:05:48.038732 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:48.038693    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"]
Sep 09 11:05:48.072671 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:48.072643    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:05:49.809812 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:49.809785    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:05:56.286743 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:05:56.286701    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:06:01.180588 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:01.180560    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 11:06:01.184140 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:01.184116    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 11:06:01.186843 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:01.186824    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 11:06:01.190997 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:01.190981    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs8qn_8b797865-8296-4715-b5cb-69bb3f884b62/ovn-acl-logging/0.log"
Sep 09 11:06:03.058230 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.058169    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v" podUID="fb1c68b3-89fc-46f9-b147-f23fe06e1b73" containerName="main" containerID="cri-o://79d6fbe5e754264bdf19f6a75894cdc6f881283038a8a8cacf790523fd7f22aa" gracePeriod=45
Sep 09 11:06:03.308535 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.308477    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:06:03.376907 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.376873    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-tls-certs\") pod \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") "
Sep 09 11:06:03.377072 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.376958    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-home\") pod \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") "
Sep 09 11:06:03.377072 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.377006    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-model-cache\") pod \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") "
Sep 09 11:06:03.377072 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.377046    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-dshm\") pod \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") "
Sep 09 11:06:03.377220 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.377079    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-tmp-dir\") pod \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") "
Sep 09 11:06:03.377220 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.377116    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzbwr\" (UniqueName: \"kubernetes.io/projected/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-kube-api-access-wzbwr\") pod \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\" (UID: \"fb1c68b3-89fc-46f9-b147-f23fe06e1b73\") "
Sep 09 11:06:03.377333 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.377258    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-home" (OuterVolumeSpecName: "home") pod "fb1c68b3-89fc-46f9-b147-f23fe06e1b73" (UID: "fb1c68b3-89fc-46f9-b147-f23fe06e1b73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:06:03.377392 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.377315    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb1c68b3-89fc-46f9-b147-f23fe06e1b73" (UID: "fb1c68b3-89fc-46f9-b147-f23fe06e1b73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:06:03.377447 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.377382    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb1c68b3-89fc-46f9-b147-f23fe06e1b73" (UID: "fb1c68b3-89fc-46f9-b147-f23fe06e1b73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:06:03.377565 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.377548    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-model-cache\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:06:03.377624 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.377566    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-tmp-dir\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:06:03.377624 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.377574    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-home\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:06:03.379187 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.379162    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-dshm" (OuterVolumeSpecName: "dshm") pod "fb1c68b3-89fc-46f9-b147-f23fe06e1b73" (UID: "fb1c68b3-89fc-46f9-b147-f23fe06e1b73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:06:03.379296 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.379236    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-kube-api-access-wzbwr" (OuterVolumeSpecName: "kube-api-access-wzbwr") pod "fb1c68b3-89fc-46f9-b147-f23fe06e1b73" (UID: "fb1c68b3-89fc-46f9-b147-f23fe06e1b73"). InnerVolumeSpecName "kube-api-access-wzbwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:06:03.379337 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.379293    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb1c68b3-89fc-46f9-b147-f23fe06e1b73" (UID: "fb1c68b3-89fc-46f9-b147-f23fe06e1b73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:06:03.478670 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.478633    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-tls-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:06:03.478670 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.478665    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-dshm\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:06:03.478670 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.478675    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzbwr\" (UniqueName: \"kubernetes.io/projected/fb1c68b3-89fc-46f9-b147-f23fe06e1b73-kube-api-access-wzbwr\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:06:03.942852 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.942814    2570 generic.go:356] "Generic (PLEG): container finished" podID="fb1c68b3-89fc-46f9-b147-f23fe06e1b73" containerID="79d6fbe5e754264bdf19f6a75894cdc6f881283038a8a8cacf790523fd7f22aa" exitCode=0
Sep 09 11:06:03.943050 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.942865    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"
Sep 09 11:06:03.943050 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.942905    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v" event={"ID":"fb1c68b3-89fc-46f9-b147-f23fe06e1b73","Type":"ContainerDied","Data":"79d6fbe5e754264bdf19f6a75894cdc6f881283038a8a8cacf790523fd7f22aa"}
Sep 09 11:06:03.943050 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.942956    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v" event={"ID":"fb1c68b3-89fc-46f9-b147-f23fe06e1b73","Type":"ContainerDied","Data":"399295a5066398c5774f8386526b67e59d4cd681ddf06527a00acd564af0be7f"}
Sep 09 11:06:03.943050 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.942977    2570 scope.go:117] "RemoveContainer" containerID="79d6fbe5e754264bdf19f6a75894cdc6f881283038a8a8cacf790523fd7f22aa"
Sep 09 11:06:03.952070 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.952054    2570 scope.go:117] "RemoveContainer" containerID="79d6fbe5e754264bdf19f6a75894cdc6f881283038a8a8cacf790523fd7f22aa"
Sep 09 11:06:03.952350 ip-10-0-128-193 kubenswrapper[2570]: E0909 11:06:03.952332    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d6fbe5e754264bdf19f6a75894cdc6f881283038a8a8cacf790523fd7f22aa\": container with ID starting with 79d6fbe5e754264bdf19f6a75894cdc6f881283038a8a8cacf790523fd7f22aa not found: ID does not exist" containerID="79d6fbe5e754264bdf19f6a75894cdc6f881283038a8a8cacf790523fd7f22aa"
Sep 09 11:06:03.952399 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.952359    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d6fbe5e754264bdf19f6a75894cdc6f881283038a8a8cacf790523fd7f22aa"} err="failed to get container status \"79d6fbe5e754264bdf19f6a75894cdc6f881283038a8a8cacf790523fd7f22aa\": rpc error: code = NotFound desc = could not find container \"79d6fbe5e754264bdf19f6a75894cdc6f881283038a8a8cacf790523fd7f22aa\": container with ID starting with 79d6fbe5e754264bdf19f6a75894cdc6f881283038a8a8cacf790523fd7f22aa not found: ID does not exist"
Sep 09 11:06:03.967101 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.967077    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"]
Sep 09 11:06:03.971586 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:03.971564    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fe318cb6-kserve-56d545f7f5-j2z8v"]
Sep 09 11:06:04.839116 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:04.839085    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb1c68b3-89fc-46f9-b147-f23fe06e1b73" path="/var/lib/kubelet/pods/fb1c68b3-89fc-46f9-b147-f23fe06e1b73/volumes"
Sep 09 11:06:12.713664 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:12.713632    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:06:12.784700 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:12.784668    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:06:13.179776 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.179731    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"]
Sep 09 11:06:13.180177 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.180159    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1c68b3-89fc-46f9-b147-f23fe06e1b73" containerName="main"
Sep 09 11:06:13.180266 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.180180    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1c68b3-89fc-46f9-b147-f23fe06e1b73" containerName="main"
Sep 09 11:06:13.180321 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.180290    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1c68b3-89fc-46f9-b147-f23fe06e1b73" containerName="main"
Sep 09 11:06:13.183220 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.183199    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.206284 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.206261    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"]
Sep 09 11:06:13.268588 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.268562    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfkq\" (UniqueName: \"kubernetes.io/projected/dffe9fe6-04df-44be-ba97-a1e15cf8a2d6-kube-api-access-sqfkq\") pod \"maas-api-ff785c76b-5gvln\" (UID: \"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.268750 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.268636    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffe9fe6-04df-44be-ba97-a1e15cf8a2d6-metrics-certs\") pod \"maas-api-ff785c76b-5gvln\" (UID: \"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.268750 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.268694    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffe9fe6-04df-44be-ba97-a1e15cf8a2d6-maas-api-tls\") pod \"maas-api-ff785c76b-5gvln\" (UID: \"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.268901 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.268746    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffe9fe6-04df-44be-ba97-a1e15cf8a2d6-openshift-service-ca\") pod \"maas-api-ff785c76b-5gvln\" (UID: \"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.370037 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.370003    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfkq\" (UniqueName: \"kubernetes.io/projected/dffe9fe6-04df-44be-ba97-a1e15cf8a2d6-kube-api-access-sqfkq\") pod \"maas-api-ff785c76b-5gvln\" (UID: \"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.370202 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.370058    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffe9fe6-04df-44be-ba97-a1e15cf8a2d6-metrics-certs\") pod \"maas-api-ff785c76b-5gvln\" (UID: \"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.370202 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.370093    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffe9fe6-04df-44be-ba97-a1e15cf8a2d6-maas-api-tls\") pod \"maas-api-ff785c76b-5gvln\" (UID: \"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.370202 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.370132    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffe9fe6-04df-44be-ba97-a1e15cf8a2d6-openshift-service-ca\") pod \"maas-api-ff785c76b-5gvln\" (UID: \"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.370752 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.370730    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffe9fe6-04df-44be-ba97-a1e15cf8a2d6-openshift-service-ca\") pod \"maas-api-ff785c76b-5gvln\" (UID: \"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.372594 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.372571    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffe9fe6-04df-44be-ba97-a1e15cf8a2d6-maas-api-tls\") pod \"maas-api-ff785c76b-5gvln\" (UID: \"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.372721 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.372699    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffe9fe6-04df-44be-ba97-a1e15cf8a2d6-metrics-certs\") pod \"maas-api-ff785c76b-5gvln\" (UID: \"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.380956 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.380932    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfkq\" (UniqueName: \"kubernetes.io/projected/dffe9fe6-04df-44be-ba97-a1e15cf8a2d6-kube-api-access-sqfkq\") pod \"maas-api-ff785c76b-5gvln\" (UID: \"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.493926 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.493899    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:13.792672 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.792645    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"]
Sep 09 11:06:13.793597 ip-10-0-128-193 kubenswrapper[2570]: W0909 11:06:13.793574    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffe9fe6_04df_44be_ba97_a1e15cf8a2d6.slice/crio-e0168d4c7ddace876d63aa6f76b4ee56c9a1181a3d4dd7f825356e8f6a8dc11b WatchSource:0}: Error finding container e0168d4c7ddace876d63aa6f76b4ee56c9a1181a3d4dd7f825356e8f6a8dc11b: Status 404 returned error can't find the container with id e0168d4c7ddace876d63aa6f76b4ee56c9a1181a3d4dd7f825356e8f6a8dc11b
Sep 09 11:06:13.996363 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:13.996334    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln" event={"ID":"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6","Type":"ContainerStarted","Data":"e0168d4c7ddace876d63aa6f76b4ee56c9a1181a3d4dd7f825356e8f6a8dc11b"}
Sep 09 11:06:15.003439 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:15.003404    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln" event={"ID":"dffe9fe6-04df-44be-ba97-a1e15cf8a2d6","Type":"ContainerStarted","Data":"280852bf5d90d12377ccf0ec62b79d64eba79b2e592fd80e2df7c9edf6d6da05"}
Sep 09 11:06:15.003852 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:15.003469    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:15.031003 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:15.030959    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln" podStartSLOduration=1.39990203 podStartE2EDuration="2.030944059s" totalImagesPullingTime="631.042029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 11:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:06:15.030219859 +0000 UTC m=+2414.693858833" watchObservedRunningTime="2026-09-09 11:06:15.030944059 +0000 UTC m=+2414.694583033"
Sep 09 11:06:21.013372 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:21.013343    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-ff785c76b-5gvln"
Sep 09 11:06:21.114567 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:21.114535    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7998845878-vrtrh"]
Sep 09 11:06:21.114781 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:21.114723    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh" podUID="5752135e-f5b1-4ef7-a5bb-29598d19e464" containerName="maas-api" containerID="cri-o://471e7229b93ebf3e5758ebf9a620dd915e4875573ec0c176271aad7d9b96b1ea" gracePeriod=30
Sep 09 11:06:22.356366 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:22.356341    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 11:06:22.461791 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:22.461703    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6ws7\" (UniqueName: \"kubernetes.io/projected/5752135e-f5b1-4ef7-a5bb-29598d19e464-kube-api-access-s6ws7\") pod \"5752135e-f5b1-4ef7-a5bb-29598d19e464\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") "
Sep 09 11:06:22.461791 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:22.461752    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-maas-api-tls\") pod \"5752135e-f5b1-4ef7-a5bb-29598d19e464\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") "
Sep 09 11:06:22.462012 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:22.461805    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5752135e-f5b1-4ef7-a5bb-29598d19e464-openshift-service-ca\") pod \"5752135e-f5b1-4ef7-a5bb-29598d19e464\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") "
Sep 09 11:06:22.462012 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:22.461820    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-metrics-certs\") pod \"5752135e-f5b1-4ef7-a5bb-29598d19e464\" (UID: \"5752135e-f5b1-4ef7-a5bb-29598d19e464\") "
Sep 09 11:06:22.462221 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:22.462196    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5752135e-f5b1-4ef7-a5bb-29598d19e464-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5752135e-f5b1-4ef7-a5bb-29598d19e464" (UID: "5752135e-f5b1-4ef7-a5bb-29598d19e464"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 11:06:22.463841 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:22.463814    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5752135e-f5b1-4ef7-a5bb-29598d19e464" (UID: "5752135e-f5b1-4ef7-a5bb-29598d19e464"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:06:22.463918 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:22.463886    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5752135e-f5b1-4ef7-a5bb-29598d19e464-kube-api-access-s6ws7" (OuterVolumeSpecName: "kube-api-access-s6ws7") pod "5752135e-f5b1-4ef7-a5bb-29598d19e464" (UID: "5752135e-f5b1-4ef7-a5bb-29598d19e464"). InnerVolumeSpecName "kube-api-access-s6ws7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:06:22.463976 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:22.463946    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5752135e-f5b1-4ef7-a5bb-29598d19e464" (UID: "5752135e-f5b1-4ef7-a5bb-29598d19e464"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:06:22.562800 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:22.562749    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-maas-api-tls\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:06:22.562800 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:22.562799    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5752135e-f5b1-4ef7-a5bb-29598d19e464-openshift-service-ca\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:06:22.562982 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:22.562809    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5752135e-f5b1-4ef7-a5bb-29598d19e464-metrics-certs\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:06:22.562982 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:22.562818    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6ws7\" (UniqueName: \"kubernetes.io/projected/5752135e-f5b1-4ef7-a5bb-29598d19e464-kube-api-access-s6ws7\") on node \"ip-10-0-128-193.ec2.internal\" DevicePath \"\""
Sep 09 11:06:23.042372 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:23.042296    2570 generic.go:356] "Generic (PLEG): container finished" podID="5752135e-f5b1-4ef7-a5bb-29598d19e464" containerID="471e7229b93ebf3e5758ebf9a620dd915e4875573ec0c176271aad7d9b96b1ea" exitCode=0
Sep 09 11:06:23.042372 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:23.042344    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh"
Sep 09 11:06:23.042372 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:23.042357    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh" event={"ID":"5752135e-f5b1-4ef7-a5bb-29598d19e464","Type":"ContainerDied","Data":"471e7229b93ebf3e5758ebf9a620dd915e4875573ec0c176271aad7d9b96b1ea"}
Sep 09 11:06:23.042623 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:23.042409    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7998845878-vrtrh" event={"ID":"5752135e-f5b1-4ef7-a5bb-29598d19e464","Type":"ContainerDied","Data":"87140ee77830ba257cccac5f63d564cb8bff08905bc5aa30af5980b3b4e6bef2"}
Sep 09 11:06:23.042623 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:23.042442    2570 scope.go:117] "RemoveContainer" containerID="471e7229b93ebf3e5758ebf9a620dd915e4875573ec0c176271aad7d9b96b1ea"
Sep 09 11:06:23.051193 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:23.051164    2570 scope.go:117] "RemoveContainer" containerID="471e7229b93ebf3e5758ebf9a620dd915e4875573ec0c176271aad7d9b96b1ea"
Sep 09 11:06:23.051436 ip-10-0-128-193 kubenswrapper[2570]: E0909 11:06:23.051412    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471e7229b93ebf3e5758ebf9a620dd915e4875573ec0c176271aad7d9b96b1ea\": container with ID starting with 471e7229b93ebf3e5758ebf9a620dd915e4875573ec0c176271aad7d9b96b1ea not found: ID does not exist" containerID="471e7229b93ebf3e5758ebf9a620dd915e4875573ec0c176271aad7d9b96b1ea"
Sep 09 11:06:23.051499 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:23.051448    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471e7229b93ebf3e5758ebf9a620dd915e4875573ec0c176271aad7d9b96b1ea"} err="failed to get container status \"471e7229b93ebf3e5758ebf9a620dd915e4875573ec0c176271aad7d9b96b1ea\": rpc error: code = NotFound desc = could not find container \"471e7229b93ebf3e5758ebf9a620dd915e4875573ec0c176271aad7d9b96b1ea\": container with ID starting with 471e7229b93ebf3e5758ebf9a620dd915e4875573ec0c176271aad7d9b96b1ea not found: ID does not exist"
Sep 09 11:06:23.070435 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:23.070405    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7998845878-vrtrh"]
Sep 09 11:06:23.072426 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:23.072408    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7998845878-vrtrh"]
Sep 09 11:06:24.839140 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:24.839113    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5752135e-f5b1-4ef7-a5bb-29598d19e464" path="/var/lib/kubelet/pods/5752135e-f5b1-4ef7-a5bb-29598d19e464/volumes"
Sep 09 11:06:53.627118 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:53.627076    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68dd87c4cd-66t9l_949317c2-d4da-4ef6-9ee4-68a565088b20/manager/0.log"
Sep 09 11:06:54.014530 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:54.014500    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-544cbf49fb-2lx89_cde22cf6-342a-438c-948a-be71c8ad26c3/copy-manifests/0.log"
Sep 09 11:06:54.029018 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:54.028992    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-544cbf49fb-2lx89_cde22cf6-342a-438c-948a-be71c8ad26c3/manager/0.log"
Sep 09 11:06:54.690183 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:54.690153    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-ff785c76b-5gvln_dffe9fe6-04df-44be-ba97-a1e15cf8a2d6/maas-api/0.log"
Sep 09 11:06:55.170789 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:55.170726    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-8zz26_e4139afe-da52-483e-b181-05c72f8299d4/postgres/0.log"
Sep 09 11:06:56.094962 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:56.094931    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-f54rk_e17d4ac4-8ae3-4cee-b448-bc97ee4e57ba/manager/0.log"
Sep 09 11:06:56.221217 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:56.221190    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-5kmnp_4404cd16-87dd-4f1a-9ba6-65fa0a3cc57f/manager/0.log"
Sep 09 11:06:56.334298 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:56.334272    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-j2lmx_0aca6679-454a-4d6d-9f65-e4deda62852c/kuadrant-console-plugin/0.log"
Sep 09 11:06:57.583689 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:57.583663    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lv97z_cd184f91-b4c9-4291-9056-e6529b1f8541/istio-proxy/1.log"
Sep 09 11:06:57.693983 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:57.693957    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-jqmwb_0fb7e67c-33c8-4efd-9b54-41eae6add320/payload-pre-processing/0.log"
Sep 09 11:06:57.947915 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:57.947841    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75db9dfd44-b2ptj_c824a8cb-67bf-4ee7-bc76-919452663ea1/router/0.log"
Sep 09 11:06:58.294397 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:58.294372    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mml5b_846ad5fa-5f7f-4478-84c2-51884387ac91/main/0.log"
Sep 09 11:06:58.520832 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:58.520807    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-2dp26_0e4ba653-6a69-45b3-9c10-b7502cbeb628/main/0.log"
Sep 09 11:06:58.744700 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:58.744678    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvpbpz_828e34e7-50dc-4cd4-a160-e24b54870701/main/0.log"
Sep 09 11:06:58.864205 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:06:58.864179    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-4v46d_700b8066-3b32-437a-bd61-3a400c6a1229/main/0.log"
Sep 09 11:07:04.481181 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:04.481116    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:07:04.658780 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:04.658734    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log" limit=1024
Sep 09 11:07:08.024664 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:08.024635    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-crdlr_11460f81-a3fc-4a03-aa3f-79dc61b95a4a/global-pull-secret-syncer/0.log"
Sep 09 11:07:08.132491 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:08.132464    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4t6qk_9fd12634-040b-49e0-b16b-30c8ac321311/konnectivity-agent/0.log"
Sep 09 11:07:08.234145 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:08.234119    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-193.ec2.internal_046071a0f74eb379e298ca5ffe8a80f1/haproxy/0.log"
Sep 09 11:07:12.833230 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:12.833197    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-f54rk_e17d4ac4-8ae3-4cee-b448-bc97ee4e57ba/manager/0.log"
Sep 09 11:07:12.862850 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:12.862782    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-5kmnp_4404cd16-87dd-4f1a-9ba6-65fa0a3cc57f/manager/0.log"
Sep 09 11:07:12.904256 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:12.904235    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-j2lmx_0aca6679-454a-4d6d-9f65-e4deda62852c/kuadrant-console-plugin/0.log"
Sep 09 11:07:16.145257 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:16.145230    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tz5nl_6d154fb6-5d37-4013-b8e1-da86a5a410b6/node-exporter/0.log"
Sep 09 11:07:16.215745 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:16.215719    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tz5nl_6d154fb6-5d37-4013-b8e1-da86a5a410b6/kube-rbac-proxy/0.log"
Sep 09 11:07:16.283781 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:16.283751    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tz5nl_6d154fb6-5d37-4013-b8e1-da86a5a410b6/init-textfile/0.log"
Sep 09 11:07:16.497414 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:16.497370    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a32f131-bb91-4a70-a75c-5873fcc41367/prometheus/0.log"
Sep 09 11:07:16.555075 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:16.555003    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a32f131-bb91-4a70-a75c-5873fcc41367/config-reloader/0.log"
Sep 09 11:07:16.601243 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:16.601204    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a32f131-bb91-4a70-a75c-5873fcc41367/thanos-sidecar/0.log"
Sep 09 11:07:16.634954 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:16.634932    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a32f131-bb91-4a70-a75c-5873fcc41367/kube-rbac-proxy-web/0.log"
Sep 09 11:07:16.674684 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:16.674664    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a32f131-bb91-4a70-a75c-5873fcc41367/kube-rbac-proxy/0.log"
Sep 09 11:07:16.714712 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:16.714689    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a32f131-bb91-4a70-a75c-5873fcc41367/kube-rbac-proxy-thanos/0.log"
Sep 09 11:07:16.745215 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:16.745196    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a32f131-bb91-4a70-a75c-5873fcc41367/init-config-reloader/0.log"
Sep 09 11:07:16.794879 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:16.794856    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-6mjnt_e3923f82-5f6f-4583-87fb-2b640355eafe/prometheus-operator/0.log"
Sep 09 11:07:16.834844 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:16.834787    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-6mjnt_e3923f82-5f6f-4583-87fb-2b640355eafe/kube-rbac-proxy/0.log"
Sep 09 11:07:17.433152 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.433125    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"]
Sep 09 11:07:17.433512 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.433470    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5752135e-f5b1-4ef7-a5bb-29598d19e464" containerName="maas-api"
Sep 09 11:07:17.433512 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.433482    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5752135e-f5b1-4ef7-a5bb-29598d19e464" containerName="maas-api"
Sep 09 11:07:17.433590 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.433544    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5752135e-f5b1-4ef7-a5bb-29598d19e464" containerName="maas-api"
Sep 09 11:07:17.436425 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.436406    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.439376 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.439359    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-86rrh\"/\"kube-root-ca.crt\""
Sep 09 11:07:17.440525 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.440508    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-86rrh\"/\"default-dockercfg-l6fff\""
Sep 09 11:07:17.440609 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.440512    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-86rrh\"/\"openshift-service-ca.crt\""
Sep 09 11:07:17.449974 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.449956    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"]
Sep 09 11:07:17.539981 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.539954    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8eb95b87-a05d-47b5-af11-95f74d9921a8-lib-modules\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.540089 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.539997    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8eb95b87-a05d-47b5-af11-95f74d9921a8-sys\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.540089 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.540014    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8eb95b87-a05d-47b5-af11-95f74d9921a8-podres\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.540165 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.540114    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8eb95b87-a05d-47b5-af11-95f74d9921a8-proc\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.540165 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.540154    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft89x\" (UniqueName: \"kubernetes.io/projected/8eb95b87-a05d-47b5-af11-95f74d9921a8-kube-api-access-ft89x\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.640892 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.640862    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8eb95b87-a05d-47b5-af11-95f74d9921a8-sys\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.641031 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.640904    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8eb95b87-a05d-47b5-af11-95f74d9921a8-podres\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.641031 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.640965    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8eb95b87-a05d-47b5-af11-95f74d9921a8-proc\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.641031 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.640982    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8eb95b87-a05d-47b5-af11-95f74d9921a8-sys\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.641031 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.640995    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft89x\" (UniqueName: \"kubernetes.io/projected/8eb95b87-a05d-47b5-af11-95f74d9921a8-kube-api-access-ft89x\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.641221 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.641075    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8eb95b87-a05d-47b5-af11-95f74d9921a8-lib-modules\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.641221 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.641091    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8eb95b87-a05d-47b5-af11-95f74d9921a8-podres\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.641221 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.641093    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8eb95b87-a05d-47b5-af11-95f74d9921a8-proc\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.641221 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.641215    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8eb95b87-a05d-47b5-af11-95f74d9921a8-lib-modules\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.652331 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.652299    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft89x\" (UniqueName: \"kubernetes.io/projected/8eb95b87-a05d-47b5-af11-95f74d9921a8-kube-api-access-ft89x\") pod \"perf-node-gather-daemonset-9cjbz\" (UID: \"8eb95b87-a05d-47b5-af11-95f74d9921a8\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:17.745988 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:17.745963    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:18.021164 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:18.021137    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"]
Sep 09 11:07:18.022013 ip-10-0-128-193 kubenswrapper[2570]: W0909 11:07:18.021986    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8eb95b87_a05d_47b5_af11_95f74d9921a8.slice/crio-1212aa34b18b917ed745b0a69cfa92f66e1ca56e54c4bf5d18f5b558012e6b63 WatchSource:0}: Error finding container 1212aa34b18b917ed745b0a69cfa92f66e1ca56e54c4bf5d18f5b558012e6b63: Status 404 returned error can't find the container with id 1212aa34b18b917ed745b0a69cfa92f66e1ca56e54c4bf5d18f5b558012e6b63
Sep 09 11:07:18.302538 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:18.302464    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz" event={"ID":"8eb95b87-a05d-47b5-af11-95f74d9921a8","Type":"ContainerStarted","Data":"e653cdb40f079cd0fa2bc43a9976c72fc2ff9e49ec06e44a3e6f48d9a73fdc19"}
Sep 09 11:07:18.302538 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:18.302510    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz" event={"ID":"8eb95b87-a05d-47b5-af11-95f74d9921a8","Type":"ContainerStarted","Data":"1212aa34b18b917ed745b0a69cfa92f66e1ca56e54c4bf5d18f5b558012e6b63"}
Sep 09 11:07:18.302704 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:18.302553    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:18.320869 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:18.320831    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz" podStartSLOduration=1.320817688 podStartE2EDuration="1.320817688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 11:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:07:18.320040776 +0000 UTC m=+2477.983679751" watchObservedRunningTime="2026-09-09 11:07:18.320817688 +0000 UTC m=+2477.984456664"
Sep 09 11:07:19.245747 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:19.245720    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/1.log"
Sep 09 11:07:19.250164 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:19.250143    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-67lpw_e16b6c9f-7090-4e95-b0eb-0b07907d992f/console-operator/2.log"
Sep 09 11:07:21.743989 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:21.743963    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-75ncn_1c1c0fa9-ce40-473e-a699-540afd0a9ed9/dns/0.log"
Sep 09 11:07:21.771010 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:21.770987    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-75ncn_1c1c0fa9-ce40-473e-a699-540afd0a9ed9/kube-rbac-proxy/0.log"
Sep 09 11:07:21.917375 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:21.917347    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dh892_935d141b-8ba9-42a3-9a46-91f75e24a0bf/dns-node-resolver/0.log"
Sep 09 11:07:22.569200 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:22.569152    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-687d48f8b6-jzq2h_720cdc3c-d15f-4788-99e6-ab6c465ff027/registry/0.log"
Sep 09 11:07:22.608263 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:22.608240    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5j47k_55711478-e733-4bdb-a947-0b6f37cc3f75/node-ca/0.log"
Sep 09 11:07:24.188950 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:24.188916    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lv97z_cd184f91-b4c9-4291-9056-e6529b1f8541/istio-proxy/0.log"
Sep 09 11:07:24.195914 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:24.195892    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lv97z_cd184f91-b4c9-4291-9056-e6529b1f8541/istio-proxy/1.log"
Sep 09 11:07:24.256251 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:24.256226    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-jqmwb_0fb7e67c-33c8-4efd-9b54-41eae6add320/payload-pre-processing/0.log"
Sep 09 11:07:24.317467 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:24.317441    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-9cjbz"
Sep 09 11:07:24.421615 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:24.421575    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75db9dfd44-b2ptj_c824a8cb-67bf-4ee7-bc76-919452663ea1/router/0.log"
Sep 09 11:07:25.465455 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:25.465420    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5llp7_6d95d07c-cf59-43f5-a825-a0744b99eafb/serve-healthcheck-canary/0.log"
Sep 09 11:07:26.353752 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:26.353659    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qddgs_133066fe-322b-44f6-85f7-3e6f43840330/kube-rbac-proxy/0.log"
Sep 09 11:07:26.386518 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:26.386492    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qddgs_133066fe-322b-44f6-85f7-3e6f43840330/exporter/0.log"
Sep 09 11:07:26.416381 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:26.416349    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qddgs_133066fe-322b-44f6-85f7-3e6f43840330/extractor/0.log"
Sep 09 11:07:28.562505 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:28.562471    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lv97z_cd184f91-b4c9-4291-9056-e6529b1f8541/istio-proxy/0.log"
Sep 09 11:07:28.569733 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:28.569713    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lv97z_cd184f91-b4c9-4291-9056-e6529b1f8541/istio-proxy/1.log"
Sep 09 11:07:28.596076 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:28.596053    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-jqmwb_0fb7e67c-33c8-4efd-9b54-41eae6add320/payload-pre-processing/0.log"
Sep 09 11:07:28.650978 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:28.650954    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75db9dfd44-b2ptj_c824a8cb-67bf-4ee7-bc76-919452663ea1/router/0.log"
Sep 09 11:07:29.419828 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:29.419792    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68dd87c4cd-66t9l_949317c2-d4da-4ef6-9ee4-68a565088b20/manager/0.log"
Sep 09 11:07:29.610407 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:29.610378    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-544cbf49fb-2lx89_cde22cf6-342a-438c-948a-be71c8ad26c3/manager/0.log"
Sep 09 11:07:29.668666 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:29.668639    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-544cbf49fb-2lx89_cde22cf6-342a-438c-948a-be71c8ad26c3/copy-manifests/0.log"
Sep 09 11:07:30.494432 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:30.494400    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lv97z_cd184f91-b4c9-4291-9056-e6529b1f8541/istio-proxy/0.log"
Sep 09 11:07:30.502196 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:30.502169    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lv97z_cd184f91-b4c9-4291-9056-e6529b1f8541/istio-proxy/1.log"
Sep 09 11:07:30.545935 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:30.545892    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-jqmwb_0fb7e67c-33c8-4efd-9b54-41eae6add320/payload-pre-processing/0.log"
Sep 09 11:07:30.668158 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:30.668131    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75db9dfd44-b2ptj_c824a8cb-67bf-4ee7-bc76-919452663ea1/router/0.log"
Sep 09 11:07:31.322174 ip-10-0-128-193 kubenswrapper[2570]: I0909 11:07:31.322146    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7658745f98-zpbnk_5c93c33a-2f17-4516-b59d-d639733c89bd/manager/0.log"