Sep 16 12:03:23.139867 ip-10-0-136-98 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 12:03:23.491051 ip-10-0-136-98 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:03:23.491051 ip-10-0-136-98 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 12:03:23.491051 ip-10-0-136-98 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:03:23.491051 ip-10-0-136-98 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 12:03:23.492500 ip-10-0-136-98 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:03:23.493356 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.493266    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 12:03:23.499829 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499807    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:23.499829 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499824    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:23.499829 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499828    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:23.499829 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499831    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:23.499829 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499834    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:23.499829 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499838    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499841    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499844    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499846    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499849    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499852    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499855    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499858    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499862    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499864    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499867    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499871    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499874    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499876    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499879    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499882    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499884    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499888    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499893    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499896    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:23.500031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499899    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499902    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499904    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499907    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499909    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499912    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499914    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499918    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499921    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499923    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499926    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499928    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499931    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499934    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499937    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499939    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499942    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499945    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499948    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:23.500516 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499951    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499954    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499958    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499961    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499964    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499967    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499972    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499976    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499979    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499982    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499984    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499987    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499990    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499992    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499995    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.499998    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500000    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500003    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500007    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:23.501002 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500010    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500013    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500015    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500018    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500021    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500023    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500026    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500030    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500033    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500036    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500038    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500041    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500044    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500046    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500049    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500052    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500054    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500057    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500059    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500062    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:23.501463 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500065    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500068    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500070    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500073    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500075    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500078    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500081    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500083    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500086    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500088    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500099    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500102    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500110    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500113    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500115    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500118    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500847    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500854    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500857    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500860    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:23.501984 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500863    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500866    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500869    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500871    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500874    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500877    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500879    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500882    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500885    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500887    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500899    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500902    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500905    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500908    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500911    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500913    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500916    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500919    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500922    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500925    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:23.502461 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500928    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500930    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500933    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500936    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500939    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500941    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500944    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500946    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500949    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500952    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500954    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500957    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500961    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500963    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500966    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500969    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500972    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500974    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500977    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500980    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:23.502975 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500982    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500985    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500988    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500991    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500993    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500996    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.500999    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501002    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501005    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501008    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501011    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501013    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501016    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501020    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501022    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501025    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501028    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501030    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501033    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:23.503454 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501035    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501038    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501040    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501043    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501045    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501048    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501051    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501053    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501056    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501058    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501061    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501063    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501066    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501068    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501071    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501074    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501076    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501079    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501082    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:23.503940 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501091    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501096    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501099    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501101    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501106    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501109    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501112    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501114    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501117    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501119    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501122    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501126    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501130    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501132    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501135    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501138    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501142    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501511    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501516    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:23.504405 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501519    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501522    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501525    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501528    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501530    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501533    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501536    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501538    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501541    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501558    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501560    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501563    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501566    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501568    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501574    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501577    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501579    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501584    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501586    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501589    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501591    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:23.504902 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501594    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501597    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501599    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501602    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501605    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501607    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501610    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501612    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501616    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501619    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501623    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501625    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501629    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501633    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501636    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501639    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501642    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501645    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501648    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:23.505418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501650    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501653    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501655    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501657    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501660    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501663    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501666    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501668    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501672    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501676    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501679    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501681    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501684    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501686    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501689    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501691    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501694    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501696    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501699    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:23.505905 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501702    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501705    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501707    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501709    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501712    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501714    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501717    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501720    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501722    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501725    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501727    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501730    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501732    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501736    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501738    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501741    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501744    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501746    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501749    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501751    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:23.506364 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501754    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501758    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501762    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501765    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501767    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501771    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501774    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501779    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501782    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501784    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501787    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501790    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501793    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501796    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501798    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501801    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501803    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.501805    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501877    2558 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 12:03:23.506917 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501914    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501920    2558 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501925    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501929    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501932    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501937    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501941    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501944    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501947    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501951    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501954    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501957    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501960    2558 flags.go:64] FLAG: --cgroup-root=""
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501963    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501967    2558 flags.go:64] FLAG: --client-ca-file=""
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501971    2558 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501975    2558 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501978    2558 flags.go:64] FLAG: --cluster-domain=""
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501981    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501984    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501987    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501991    2558 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501995    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 12:03:23.507369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.501998    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502002    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502005    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502008    2558 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502011    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502014    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502017    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502020    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502023    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502027    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502030    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502032    2558 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502036    2558 flags.go:64] FLAG: --enable-server="true"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502039    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502043    2558 flags.go:64] FLAG: --event-burst="100"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502046    2558 flags.go:64] FLAG: --event-qps="50"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502049    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502052    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502055    2558 flags.go:64] FLAG: --eviction-hard=""
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502058    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502061    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502064    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502067    2558 flags.go:64] FLAG: --eviction-soft=""
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502071    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502075    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 12:03:23.507930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502079    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502082    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502085    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502088    2558 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502091    2558 flags.go:64] FLAG: --feature-gates=""
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502094    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502098    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502101    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502104    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502107    2558 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502110    2558 flags.go:64] FLAG: --help="false"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502113    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502116    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502119    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502122    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502125    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502128    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502131    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502134    2558 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502136    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502139    2558 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502142    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502145    2558 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502148    2558 flags.go:64] FLAG: --kube-reserved=""
Sep 16 12:03:23.508560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502151    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502154    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502157    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502159    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502162    2558 flags.go:64] FLAG: --lock-file=""
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502165    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502170    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502174    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502180    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502183    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502186    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502189    2558 flags.go:64] FLAG: --logging-format="text"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502192    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502195    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502198    2558 flags.go:64] FLAG: --manifest-url=""
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502201    2558 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502206    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502209    2558 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502213    2558 flags.go:64] FLAG: --max-pods="110"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502215    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502218    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502221    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502224    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502227    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502230    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 12:03:23.509154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502233    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502240    2558 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502243    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502246    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502249    2558 flags.go:64] FLAG: --pod-cidr=""
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502252    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502257    2558 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502260    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502263    2558 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502266    2558 flags.go:64] FLAG: --port="10250"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502269    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502272    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00c37ede94a45565d"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502275    2558 flags.go:64] FLAG: --qos-reserved=""
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502280    2558 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502283    2558 flags.go:64] FLAG: --register-node="true"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502287    2558 flags.go:64] FLAG: --register-with-taints=""
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502290    2558 flags.go:64] FLAG: --registry-burst="10"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502294    2558 flags.go:64] FLAG: --registry-qps="5"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502297    2558 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502299    2558 flags.go:64] FLAG: --reserved-memory=""
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502303    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502306    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502309    2558 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502312    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502314    2558 flags.go:64] FLAG: --runonce="false"
Sep 16 12:03:23.509790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502317    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502320    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502323    2558 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502326    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502329    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502332    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502335    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502338    2558 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502340    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502343    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502346    2558 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502349    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502351    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502354    2558 flags.go:64] FLAG: --system-cgroups=""
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502357    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502362    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502365    2558 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502368    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502372    2558 flags.go:64] FLAG: --tls-min-version=""
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502374    2558 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502378    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502381    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502385    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502388    2558 flags.go:64] FLAG: --v="2"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502392    2558 flags.go:64] FLAG: --version="false"
Sep 16 12:03:23.510425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502396    2558 flags.go:64] FLAG: --vmodule=""
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502400    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502403    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502527    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502531    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502535    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502538    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502541    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502556    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502560    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502562    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502567    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502570    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502575    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502579    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502582    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502585    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502588    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502591    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:23.511132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502594    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502597    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502600    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502602    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502605    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502608    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502611    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502614    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502618    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502621    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502625    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502628    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502631    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502633    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502636    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502640    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502644    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502647    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502650    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502652    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:23.511639 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502663    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502666    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502669    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502672    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502674    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502677    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502680    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502683    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502685    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502688    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502691    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502693    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502696    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502698    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502701    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502703    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502706    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502709    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502712    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502715    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:23.512134 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502718    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502721    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502724    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502726    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502729    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502731    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502734    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502737    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502740    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502742    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502745    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502747    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502750    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502753    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502755    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502758    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502760    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502763    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502765    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502768    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:23.512640 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502771    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502773    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502776    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502779    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502781    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502784    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502786    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502789    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502792    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502794    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502797    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502799    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502803    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502806    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502809    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502811    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502814    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502816    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502819    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:23.513136 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502821    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502824    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502827    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.502829    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.502837    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.509938    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.509955    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.509975    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.509981    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510054    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510059    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510062    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510065    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510068    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510070    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:23.513621 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510073    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510076    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510079    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510081    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510084    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510086    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510089    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510091    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510094    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510097    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510100    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510102    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510105    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510108    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510111    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510113    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510116    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510119    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510121    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510124    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:23.514031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510126    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510129    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510132    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510136    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510141    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510144    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510147    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510150    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510152    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510154    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510157    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510160    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510162    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510165    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510167    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510170    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510172    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510175    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510177    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:23.514519 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510180    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510183    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510186    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510188    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510191    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510193    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510196    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510199    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510202    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510204    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510208    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510210    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510213    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510215    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510218    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510222    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510224    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510227    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510230    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510233    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:23.514996 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510235    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510238    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510240    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510243    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510245    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510248    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510250    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510253    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510255    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510258    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510260    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510263    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510266    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510268    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510271    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510274    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510276    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510279    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510281    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510284    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:23.515509 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510286    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510289    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510292    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510294    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510297    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510300    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510302    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510305    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510307    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510312    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510315    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510318    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510322    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510326    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:23.516016 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.510331    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510473    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510478    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510482    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510486    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510489    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510492    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510495    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510498    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510501    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510505    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510508    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510511    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510514    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510516    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510519    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510522    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510524    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510527    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:23.516381 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510530    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510532    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510535    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510537    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510540    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510557    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510561    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510564    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510567    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510569    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510579    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510581    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510584    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510587    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510589    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510592    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510594    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510597    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510600    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510602    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:23.516869 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510605    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510607    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510609    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510612    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510614    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510617    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510619    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510622    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510625    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510627    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510630    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510632    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510635    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510638    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510640    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510643    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510645    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510648    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510650    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510653    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:23.517348 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510656    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510658    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510661    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510669    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510672    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510675    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510677    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510680    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510682    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510685    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510687    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510690    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510692    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510695    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510697    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510700    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510702    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510705    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510708    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:23.517952 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510711    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510713    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510716    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510718    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510720    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510723    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510726    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510728    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510731    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510733    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510735    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510738    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510741    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510743    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510745    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510748    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510750    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510760    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510763    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:23.518418 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510766    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:23.518903 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510769    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:23.518903 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:23.510772    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:23.518903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.510777    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:03:23.518903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.510908    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 12:03:23.518903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.515160    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 12:03:23.518903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.516074    2558 server.go:1019] "Starting client certificate rotation"
Sep 16 12:03:23.518903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.516175    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:03:23.518903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.516202    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:03:23.518903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.518748    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:03:23.524052 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.524028    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:03:23.532830 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.532813    2558 log.go:25] "Validated CRI v1 runtime API"
Sep 16 12:03:23.535968 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.535950    2558 log.go:25] "Validated CRI v1 image API"
Sep 16 12:03:23.536778 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.536763    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 12:03:23.542337 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.542319    2558 fs.go:135] Filesystem UUIDs: map[1dcbc034-1aa5-497b-af1f-1ff79d11415c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a622854d-356a-4346-a0b4-7256d9e15022:/dev/nvme0n1p4]
Sep 16 12:03:23.542392 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.542337    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 12:03:23.548026 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.547911    2558 manager.go:216] Machine: {Timestamp:2026-09-16 12:03:23.5465776 +0000 UTC m=+0.300536328 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099803 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22525002b948b8e4082aae499e4ba6 SystemUUID:ec225250-02b9-48b8-e408-2aae499e4ba6 BootID:57b51fde-2f33-4133-a5a5-9067945bf022 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:3a:d3:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:3a:d3:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:2b:8e:92:b8:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 12:03:23.548026 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.548017    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 12:03:23.548196 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.548107    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 12:03:23.549979 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.549964    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:03:23.551002 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.550974    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 12:03:23.551185 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.551005    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-98.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 12:03:23.551831 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.551821    2558 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 12:03:23.551864 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.551833    2558 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 12:03:23.551864 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.551840    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:03:23.552466 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.552451    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:03:23.553533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.553524    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 12:03:23.557673 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.557662    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:03:23.557781 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.557773    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 12:03:23.557818 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.557801    2558 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 12:03:23.557818 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.557811    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 12:03:23.557869 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.557823    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 12:03:23.557869 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.557833    2558 kubelet.go:398] "Adding apiserver pod source"
Sep 16 12:03:23.557869 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.557842    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 12:03:23.558288 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.558276    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:03:23.558318 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.558300    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:03:23.561276 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.561258    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 12:03:23.563951 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.563937    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 12:03:23.563999 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.563969    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 12:03:23.565780 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565763    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 12:03:23.565780 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565779    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 12:03:23.565780 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565786    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 12:03:23.565903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565791    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 12:03:23.565903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565800    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 12:03:23.565903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565806    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 12:03:23.565903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565820    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 12:03:23.565903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565825    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 12:03:23.565903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565831    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 12:03:23.565903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565837    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 12:03:23.565903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565845    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 12:03:23.565903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565851    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 12:03:23.565903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565873    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 12:03:23.565903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.565879    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 12:03:23.568559 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.568534    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 12:03:23.568770 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.568746    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 12:03:23.568822 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.568798    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 12:03:23.571002 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.570983    2558 server.go:1268] "Started kubelet"
Sep 16 12:03:23.571096 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.571024    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 12:03:23.571157 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.571113    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 12:03:23.571211 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.571181    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 12:03:23.572326 ip-10-0-136-98 systemd[1]: Started Kubernetes Kubelet.
Sep 16 12:03:23.572511 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.572450    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 12:03:23.572990 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.572920    2558 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 12:03:23.578122 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.578100    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 12:03:23.578384 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.577512    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-98.ec2.internal.18d5cb860e05bfa4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-98.ec2.internal,UID:ip-10-0-136-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-98.ec2.internal,},FirstTimestamp:2026-09-16 12:03:23.570954148 +0000 UTC m=+0.324912871,LastTimestamp:2026-09-16 12:03:23.570954148 +0000 UTC m=+0.324912871,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-98.ec2.internal,}"
Sep 16 12:03:23.580560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.580531    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 12:03:23.581080 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.581066    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 12:03:23.581183 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.581162    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-98.ec2.internal\" not found"
Sep 16 12:03:23.581883 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.581761    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 12:03:23.581883 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.581784    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 12:03:23.581883 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.581799    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 12:03:23.581883 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.581879    2558 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 12:03:23.581883 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.581889    2558 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 12:03:23.582881 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.582864    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 12:03:23.582955 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.582883    2558 factory.go:55] Registering systemd factory
Sep 16 12:03:23.582955 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.582893    2558 factory.go:223] Registration of the systemd container factory successfully
Sep 16 12:03:23.583137 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.583118    2558 factory.go:153] Registering CRI-O factory
Sep 16 12:03:23.583137 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.583136    2558 factory.go:223] Registration of the crio container factory successfully
Sep 16 12:03:23.583237 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.583155    2558 factory.go:103] Registering Raw factory
Sep 16 12:03:23.583237 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.583181    2558 manager.go:1196] Started watching for new ooms in manager
Sep 16 12:03:23.583510 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.583498    2558 manager.go:349] Starting recovery of all containers
Sep 16 12:03:23.592395 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.592364    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 12:03:23.592395 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.592382    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 12:03:23.594318 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.594302    2558 manager.go:354] Recovery completed
Sep 16 12:03:23.600282 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.600256    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5sfxn"
Sep 16 12:03:23.601121 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.601108    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:23.603115 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.603096    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:23.603200 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.603134    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:23.603200 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.603144    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:23.603761 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.603744    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 12:03:23.603761 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.603758    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 12:03:23.603877 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.603779    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:23.603877 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.603785    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:03:23.604483 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.604467    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:23.604539 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.604492    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:23.604539 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.604502    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:23.608393 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.608262    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5sfxn"
Sep 16 12:03:23.613912 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.613849    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-98.ec2.internal.18d5cb860ff0737f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-98.ec2.internal,UID:ip-10-0-136-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-98.ec2.internal,},FirstTimestamp:2026-09-16 12:03:23.603112831 +0000 UTC m=+0.357071552,LastTimestamp:2026-09-16 12:03:23.603112831 +0000 UTC m=+0.357071552,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-98.ec2.internal,}"
Sep 16 12:03:23.623329 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.623254    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-98.ec2.internal.18d5cb860ff0daa9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-98.ec2.internal,UID:ip-10-0-136-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-98.ec2.internal,},FirstTimestamp:2026-09-16 12:03:23.603139241 +0000 UTC m=+0.357097963,LastTimestamp:2026-09-16 12:03:23.603139241 +0000 UTC m=+0.357097963,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-98.ec2.internal,}"
Sep 16 12:03:23.631267 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.631204    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-98.ec2.internal.18d5cb860ff0fe6b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-98.ec2.internal,UID:ip-10-0-136-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-98.ec2.internal,},FirstTimestamp:2026-09-16 12:03:23.603148395 +0000 UTC m=+0.357107117,LastTimestamp:2026-09-16 12:03:23.603148395 +0000 UTC m=+0.357107117,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-98.ec2.internal,}"
Sep 16 12:03:23.639666 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.639576    2558 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-98.ec2.internal.18d5cb860ff0737f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-98.ec2.internal.18d5cb860ff0737f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-98.ec2.internal,UID:ip-10-0-136-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-98.ec2.internal,},FirstTimestamp:2026-09-16 12:03:23.603112831 +0000 UTC m=+0.357071552,LastTimestamp:2026-09-16 12:03:23.604481288 +0000 UTC m=+0.358440011,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-98.ec2.internal,}"
Sep 16 12:03:23.658814 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.650714    2558 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-98.ec2.internal.18d5cb860ff0daa9\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-98.ec2.internal.18d5cb860ff0daa9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-98.ec2.internal,UID:ip-10-0-136-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-98.ec2.internal,},FirstTimestamp:2026-09-16 12:03:23.603139241 +0000 UTC m=+0.357097963,LastTimestamp:2026-09-16 12:03:23.604496891 +0000 UTC m=+0.358455612,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-98.ec2.internal,}"
Sep 16 12:03:23.658814 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.658388    2558 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-98.ec2.internal.18d5cb860ff0fe6b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-98.ec2.internal.18d5cb860ff0fe6b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-98.ec2.internal,UID:ip-10-0-136-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-98.ec2.internal,},FirstTimestamp:2026-09-16 12:03:23.603148395 +0000 UTC m=+0.357107117,LastTimestamp:2026-09-16 12:03:23.604505407 +0000 UTC m=+0.358464130,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-98.ec2.internal,}"
Sep 16 12:03:23.681509 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.681477    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-98.ec2.internal\" not found"
Sep 16 12:03:23.748081 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.747988    2558 policy_none.go:49] "None policy: Start"
Sep 16 12:03:23.748081 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.748011    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 12:03:23.748081 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.748038    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 12:03:23.749855 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.749830    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 12:03:23.750273 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.750261    2558 policy_none.go:47] "Start"
Sep 16 12:03:23.751054 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.751040    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 12:03:23.751119 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.751088    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 12:03:23.751164 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.751135    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 12:03:23.751252 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.751232    2558 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 16 12:03:23.761865 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.761849    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:23.780131 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.780113    2558 manager.go:337] "Starting Device Plugin manager"
Sep 16 12:03:23.780282 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.780157    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 12:03:23.780282 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.780169    2558 server.go:87] "Starting device plugin registration server"
Sep 16 12:03:23.780523 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.780512    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 12:03:23.780584 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.780525    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 12:03:23.780662 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.780648    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 12:03:23.780746 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.780728    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 12:03:23.780746 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.780740    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 12:03:23.781257 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.781238    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 12:03:23.781342 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.781286    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-98.ec2.internal\" not found"
Sep 16 12:03:23.798430 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.798410    2558 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-98.ec2.internal\" not found" node="ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.851541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.851509    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-98.ec2.internal"]
Sep 16 12:03:23.851669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.851647    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:23.852718 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.852698    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:23.852831 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.852736    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:23.852831 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.852751    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:23.855173 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.855160    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:23.855310 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.855287    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.855365 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.855324    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:23.855858 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.855842    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:23.855956 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.855864    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:23.855956 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.855874    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:23.855956 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.855894    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:23.855956 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.855918    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:23.855956 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.855935    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:23.858280 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.858262    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.858363 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.858294    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:23.858934 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.858918    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:23.859007 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.858942    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:23.859007 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.858952    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:23.880658 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.880640    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:23.881405 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.881389    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:23.881472 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.881415    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:23.881472 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.881426    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:23.881472 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.881454    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.882256 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.882240    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-98.ec2.internal\" not found" node="ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.883105 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.883090    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9288b06b268d24887bc1ba15eed675a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal\" (UID: \"9288b06b268d24887bc1ba15eed675a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.883151 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.883113    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9288b06b268d24887bc1ba15eed675a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal\" (UID: \"9288b06b268d24887bc1ba15eed675a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.883151 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.883130    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7698f8b509c65d280b6ab05a4a563a4f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-98.ec2.internal\" (UID: \"7698f8b509c65d280b6ab05a4a563a4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.886854 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.886840    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-98.ec2.internal\" not found" node="ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.889806 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.889793    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.889853 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.889811    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-98.ec2.internal\": node \"ip-10-0-136-98.ec2.internal\" not found"
Sep 16 12:03:23.929106 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:23.929080    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-98.ec2.internal\" not found"
Sep 16 12:03:23.983999 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.983973    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9288b06b268d24887bc1ba15eed675a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal\" (UID: \"9288b06b268d24887bc1ba15eed675a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.984070 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.984003    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9288b06b268d24887bc1ba15eed675a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal\" (UID: \"9288b06b268d24887bc1ba15eed675a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.984070 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.984020    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7698f8b509c65d280b6ab05a4a563a4f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-98.ec2.internal\" (UID: \"7698f8b509c65d280b6ab05a4a563a4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.984165 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.984095    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9288b06b268d24887bc1ba15eed675a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal\" (UID: \"9288b06b268d24887bc1ba15eed675a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.984165 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.984096    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9288b06b268d24887bc1ba15eed675a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal\" (UID: \"9288b06b268d24887bc1ba15eed675a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:23.984165 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:23.984101    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7698f8b509c65d280b6ab05a4a563a4f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-98.ec2.internal\" (UID: \"7698f8b509c65d280b6ab05a4a563a4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:24.029325 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:24.029260    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-98.ec2.internal\" not found"
Sep 16 12:03:24.130008 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:24.129977    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-98.ec2.internal\" not found"
Sep 16 12:03:24.185310 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.185279    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:24.190535 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.190343    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:24.230429 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:24.230404    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-98.ec2.internal\" not found"
Sep 16 12:03:24.333300 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:24.333220    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-98.ec2.internal\" not found"
Sep 16 12:03:24.398290 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.398261    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:24.435346 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:24.435324    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-98.ec2.internal\" not found"
Sep 16 12:03:24.515931 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.515903    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 12:03:24.516425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.516050    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:03:24.516425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.516067    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:03:24.536738 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:24.536714    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-98.ec2.internal\" not found"
Sep 16 12:03:24.558596 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.558577    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:24.637593 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:24.637536    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-98.ec2.internal\" not found"
Sep 16 12:03:24.659792 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.659755    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:24.661606 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.661586    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:24.661716 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.661638    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:24.661716 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.661655    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:24.709789 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.708784    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:24.752395 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.752349    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 11:58:23 +0000 UTC" deadline="2028-05-09 22:06:11.442695244 +0000 UTC"
Sep 16 12:03:24.752395 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.752389    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14434h2m46.690310966s"
Sep 16 12:03:24.781473 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.781452    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:24.794741 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.794722    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:03:24.795683 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.795669    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-98.ec2.internal"
Sep 16 12:03:24.805269 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.805250    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:03:24.807670 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:24.807626    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7698f8b509c65d280b6ab05a4a563a4f.slice/crio-3accfc44ffe282916d23a12fe4ee8f10bceded70a44871eca1d4ff5abbefd89f WatchSource:0}: Error finding container 3accfc44ffe282916d23a12fe4ee8f10bceded70a44871eca1d4ff5abbefd89f: Status 404 returned error can't find the container with id 3accfc44ffe282916d23a12fe4ee8f10bceded70a44871eca1d4ff5abbefd89f
Sep 16 12:03:24.813819 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:24.813803    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:03:24.821208 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:24.821179    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9288b06b268d24887bc1ba15eed675a9.slice/crio-66bf8b0292a304fd0468f8c45c84270d7d98497c7afcdb1248ff8b536557cf6c WatchSource:0}: Error finding container 66bf8b0292a304fd0468f8c45c84270d7d98497c7afcdb1248ff8b536557cf6c: Status 404 returned error can't find the container with id 66bf8b0292a304fd0468f8c45c84270d7d98497c7afcdb1248ff8b536557cf6c
Sep 16 12:03:25.519329 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.519294    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:25.559456 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.559424    2558 apiserver.go:52] "Watching apiserver"
Sep 16 12:03:25.567776 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.567747    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 12:03:25.569006 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.568951    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-tgspg","openshift-multus/multus-additional-cni-plugins-9tqmt","openshift-multus/multus-p8889","openshift-multus/network-metrics-daemon-pm6d7","openshift-network-operator/iptables-alerter-dhdrp","kube-system/global-pull-secret-syncer-xc9vl","kube-system/konnectivity-agent-hpmjr","openshift-cluster-node-tuning-operator/tuned-2wjkt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal","openshift-network-diagnostics/network-check-target-bzbgg","openshift-ovn-kubernetes/ovnkube-node-xflq6","kube-system/kube-apiserver-proxy-ip-10-0-136-98.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv","openshift-dns/node-resolver-fk9r5"]
Sep 16 12:03:25.571979 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.571953    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tgspg"
Sep 16 12:03:25.574266 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.574242    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.575654 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.575548    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zm2xc\""
Sep 16 12:03:25.579053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.575956    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 12:03:25.579053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.577228    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xvwkp\""
Sep 16 12:03:25.579053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.577474    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 12:03:25.579053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.577829    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 12:03:25.579053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.577896    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 12:03:25.579053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.578008    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 12:03:25.579053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.578119    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 12:03:25.579053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.578174    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 12:03:25.579053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.577835    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 12:03:25.579053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.578485    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.580881 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.580859    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jchx5\""
Sep 16 12:03:25.581129 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.581112    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 12:03:25.581325 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.581309    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 12:03:25.582525 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.582504    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:25.582647 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:25.582582    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:25.583408 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.582993    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dhdrp"
Sep 16 12:03:25.585097 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.585076    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:03:25.585097 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.585076    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 12:03:25.585480 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.585458    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tkxmv\""
Sep 16 12:03:25.585712 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.585693    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 12:03:25.586370 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.585919    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:25.586370 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:25.585977    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:25.588071 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.588048    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hpmjr"
Sep 16 12:03:25.590464 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.590444    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 12:03:25.590636 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.590603    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 12:03:25.590636 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.590448    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7xhgt\""
Sep 16 12:03:25.590800 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.590784    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.593049 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.593021    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:25.593256 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:25.593215    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:25.594293 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.593941    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:03:25.598337 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.598080    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fmwkh\""
Sep 16 12:03:25.598337 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.598134    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 12:03:25.598608 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.598588    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.599034 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.598587    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb780329-b558-4913-ae68-75262032b1a1-cni-binary-copy\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.599126 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599069    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-hostroot\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.599126 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599099    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq8qr\" (UniqueName: \"kubernetes.io/projected/91c3d651-2ead-43e7-ba31-acb29b24408f-kube-api-access-lq8qr\") pod \"iptables-alerter-dhdrp\" (UID: \"91c3d651-2ead-43e7-ba31-acb29b24408f\") " pod="openshift-network-operator/iptables-alerter-dhdrp"
Sep 16 12:03:25.599224 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599133    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-multus-conf-dir\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.599224 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599174    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af7f0b87-ae82-4e3f-b10f-c85648bde428-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.599224 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599211    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-cnibin\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.599399 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599265    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tm95\" (UniqueName: \"kubernetes.io/projected/cb780329-b558-4913-ae68-75262032b1a1-kube-api-access-8tm95\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.599399 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599307    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91c3d651-2ead-43e7-ba31-acb29b24408f-iptables-alerter-script\") pod \"iptables-alerter-dhdrp\" (UID: \"91c3d651-2ead-43e7-ba31-acb29b24408f\") " pod="openshift-network-operator/iptables-alerter-dhdrp"
Sep 16 12:03:25.599399 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599340    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b-serviceca\") pod \"node-ca-tgspg\" (UID: \"4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b\") " pod="openshift-image-registry/node-ca-tgspg"
Sep 16 12:03:25.599399 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af7f0b87-ae82-4e3f-b10f-c85648bde428-cnibin\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.599612 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599406    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af7f0b87-ae82-4e3f-b10f-c85648bde428-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.599612 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599471    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-run-netns\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.599612 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599507    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:25.599612 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599561    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af7f0b87-ae82-4e3f-b10f-c85648bde428-system-cni-dir\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.599866 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599617    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c55g4\" (UniqueName: \"kubernetes.io/projected/1617d1a8-1f68-4475-88f9-f6fdb91a07db-kube-api-access-c55g4\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:25.599866 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599665    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-var-lib-cni-bin\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.599866 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599697    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb780329-b558-4913-ae68-75262032b1a1-multus-daemon-config\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.599866 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599728    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-etc-kubernetes\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.599866 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599803    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwt7k\" (UniqueName: \"kubernetes.io/projected/af7f0b87-ae82-4e3f-b10f-c85648bde428-kube-api-access-nwt7k\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.599866 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599830    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvx9t\" (UniqueName: \"kubernetes.io/projected/4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b-kube-api-access-zvx9t\") pod \"node-ca-tgspg\" (UID: \"4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b\") " pod="openshift-image-registry/node-ca-tgspg"
Sep 16 12:03:25.600171 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599874    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af7f0b87-ae82-4e3f-b10f-c85648bde428-cni-binary-copy\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.600171 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af7f0b87-ae82-4e3f-b10f-c85648bde428-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.600171 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599932    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-run-k8s-cni-cncf-io\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.600171 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599962    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b-host\") pod \"node-ca-tgspg\" (UID: \"4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b\") " pod="openshift-image-registry/node-ca-tgspg"
Sep 16 12:03:25.600171 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599965    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:03:25.600171 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.599992    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-var-lib-cni-multus\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.600171 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.600021    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-var-lib-kubelet\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.600171 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.600085    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-run-multus-certs\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.600171 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.600123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91c3d651-2ead-43e7-ba31-acb29b24408f-host-slash\") pod \"iptables-alerter-dhdrp\" (UID: \"91c3d651-2ead-43e7-ba31-acb29b24408f\") " pod="openshift-network-operator/iptables-alerter-dhdrp"
Sep 16 12:03:25.600171 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.600154    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-system-cni-dir\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.600658 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.600345    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.600969 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.600947    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-multus-cni-dir\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.601053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.600993    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-multus-socket-dir-parent\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.602142 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.601696    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 12:03:25.602142 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.601997    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 12:03:25.602142 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.602095    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xfx8j\""
Sep 16 12:03:25.602142 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.602122    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 12:03:25.602453 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.602124    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 12:03:25.602453 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.602317    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 12:03:25.602453 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.602361    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 12:03:25.602874 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.602648    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 12:03:25.602874 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.602728    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7k52n\""
Sep 16 12:03:25.602874 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.602728    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 12:03:25.602874 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.602776    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 12:03:25.603091 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.603080    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fk9r5"
Sep 16 12:03:25.606011 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.605893    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9z6zp\""
Sep 16 12:03:25.606176 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.606159    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 12:03:25.606376 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.606361    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 12:03:25.628344 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.628320    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fzd27"
Sep 16 12:03:25.633754 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.633733    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fzd27"
Sep 16 12:03:25.682784 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.682755    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 12:03:25.701587 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701372    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-kubernetes\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.701587 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701411    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/678bc08c-e65a-4d2f-8de1-5b73269448c2-hosts-file\") pod \"node-resolver-fk9r5\" (UID: \"678bc08c-e65a-4d2f-8de1-5b73269448c2\") " pod="openshift-dns/node-resolver-fk9r5"
Sep 16 12:03:25.701587 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701439    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-run-netns\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.701587 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701467    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb780329-b558-4913-ae68-75262032b1a1-multus-daemon-config\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.701587 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701493    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-etc-kubernetes\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.701587 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701557    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvx9t\" (UniqueName: \"kubernetes.io/projected/4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b-kube-api-access-zvx9t\") pod \"node-ca-tgspg\" (UID: \"4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b\") " pod="openshift-image-registry/node-ca-tgspg"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701595    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-run-k8s-cni-cncf-io\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701622    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-var-lib-cni-multus\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701650    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-sysconfig\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701669    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-run-k8s-cni-cncf-io\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701562    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-etc-kubernetes\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701672    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-sys\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701717    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-var-lib-cni-multus\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701724    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d29zz\" (UniqueName: \"kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz\") pod \"network-check-target-bzbgg\" (UID: \"cca7ed59-7144-44f8-8421-497cb96f2aae\") " pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701749    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b-host\") pod \"node-ca-tgspg\" (UID: \"4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b\") " pod="openshift-image-registry/node-ca-tgspg"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701771    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-var-lib-kubelet\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-run-multus-certs\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701813    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91c3d651-2ead-43e7-ba31-acb29b24408f-host-slash\") pod \"iptables-alerter-dhdrp\" (UID: \"91c3d651-2ead-43e7-ba31-acb29b24408f\") " pod="openshift-network-operator/iptables-alerter-dhdrp"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701835    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-systemd\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-kubelet\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701878    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-systemd-units\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701899    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-run-systemd\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.701991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701921    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-system-cni-dir\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701945    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-multus-cni-dir\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701966    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91c3d651-2ead-43e7-ba31-acb29b24408f-host-slash\") pod \"iptables-alerter-dhdrp\" (UID: \"91c3d651-2ead-43e7-ba31-acb29b24408f\") " pod="openshift-network-operator/iptables-alerter-dhdrp"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.701969    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-multus-socket-dir-parent\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702005    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-etc-openvswitch\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702024    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-multus-socket-dir-parent\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702031    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-log-socket\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-cni-bin\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702077    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-hostroot\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702079    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb780329-b558-4913-ae68-75262032b1a1-multus-daemon-config\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702098    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq8qr\" (UniqueName: \"kubernetes.io/projected/91c3d651-2ead-43e7-ba31-acb29b24408f-kube-api-access-lq8qr\") pod \"iptables-alerter-dhdrp\" (UID: \"91c3d651-2ead-43e7-ba31-acb29b24408f\") " pod="openshift-network-operator/iptables-alerter-dhdrp"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702120    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-sysctl-d\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702140    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-tuned\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702160    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-system-cni-dir\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702166    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678bc08c-e65a-4d2f-8de1-5b73269448c2-tmp-dir\") pod \"node-resolver-fk9r5\" (UID: \"678bc08c-e65a-4d2f-8de1-5b73269448c2\") " pod="openshift-dns/node-resolver-fk9r5"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702195    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-multus-cni-dir\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702209    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-var-lib-kubelet\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.702807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-run-openvswitch\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-run-multus-certs\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702262    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-hostroot\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702287    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-run-ovn\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702332    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-run-ovn-kubernetes\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702358    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b-host\") pod \"node-ca-tgspg\" (UID: \"4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b\") " pod="openshift-image-registry/node-ca-tgspg"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702366    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af7f0b87-ae82-4e3f-b10f-c85648bde428-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702391    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-cnibin\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702423    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tm95\" (UniqueName: \"kubernetes.io/projected/cb780329-b558-4913-ae68-75262032b1a1-kube-api-access-8tm95\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91c3d651-2ead-43e7-ba31-acb29b24408f-iptables-alerter-script\") pod \"iptables-alerter-dhdrp\" (UID: \"91c3d651-2ead-43e7-ba31-acb29b24408f\") " pod="openshift-network-operator/iptables-alerter-dhdrp"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702476    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-run\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702515    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af7f0b87-ae82-4e3f-b10f-c85648bde428-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702540    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-cnibin\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702577    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a2351a4-292f-4ccd-aefc-d3467485cd0e-tmp\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702606    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzfdn\" (UniqueName: \"kubernetes.io/projected/2a2351a4-292f-4ccd-aefc-d3467485cd0e-kube-api-access-rzfdn\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702632    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b-serviceca\") pod \"node-ca-tgspg\" (UID: \"4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b\") " pod="openshift-image-registry/node-ca-tgspg"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af7f0b87-ae82-4e3f-b10f-c85648bde428-cnibin\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.703566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702685    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af7f0b87-ae82-4e3f-b10f-c85648bde428-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702710    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-run-netns\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702757    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kfx7\" (UniqueName: \"kubernetes.io/projected/678bc08c-e65a-4d2f-8de1-5b73269448c2-kube-api-access-7kfx7\") pod \"node-resolver-fk9r5\" (UID: \"678bc08c-e65a-4d2f-8de1-5b73269448c2\") " pod="openshift-dns/node-resolver-fk9r5"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702767    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af7f0b87-ae82-4e3f-b10f-c85648bde428-cnibin\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702782    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-var-lib-openvswitch\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccq9z\" (UniqueName: \"kubernetes.io/projected/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-kube-api-access-ccq9z\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702821    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-run-netns\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.702832    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af7f0b87-ae82-4e3f-b10f-c85648bde428-system-cni-dir\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:25.702944    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:25.703007    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs podName:1617d1a8-1f68-4475-88f9-f6fdb91a07db nodeName:}" failed. No retries permitted until 2026-09-16 12:03:26.202987334 +0000 UTC m=+2.956946046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs") pod "network-metrics-daemon-pm6d7" (UID: "1617d1a8-1f68-4475-88f9-f6fdb91a07db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91c3d651-2ead-43e7-ba31-acb29b24408f-iptables-alerter-script\") pod \"iptables-alerter-dhdrp\" (UID: \"91c3d651-2ead-43e7-ba31-acb29b24408f\") " pod="openshift-network-operator/iptables-alerter-dhdrp"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703086    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc-agent-certs\") pod \"konnectivity-agent-hpmjr\" (UID: \"b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc\") " pod="kube-system/konnectivity-agent-hpmjr"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703098    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b-serviceca\") pod \"node-ca-tgspg\" (UID: \"4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b\") " pod="openshift-image-registry/node-ca-tgspg"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703115    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc-konnectivity-ca\") pod \"konnectivity-agent-hpmjr\" (UID: \"b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc\") " pod="kube-system/konnectivity-agent-hpmjr"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703144    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.704379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703193    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af7f0b87-ae82-4e3f-b10f-c85648bde428-system-cni-dir\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703239    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-registration-dir\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703266    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703289    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-host\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703315    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwt7k\" (UniqueName: \"kubernetes.io/projected/af7f0b87-ae82-4e3f-b10f-c85648bde428-kube-api-access-nwt7k\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af7f0b87-ae82-4e3f-b10f-c85648bde428-cni-binary-copy\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703374    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af7f0b87-ae82-4e3f-b10f-c85648bde428-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703383    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af7f0b87-ae82-4e3f-b10f-c85648bde428-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703845    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-device-dir\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703870    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82e3f885-f60e-4039-ac7d-3318d0fb27ca-kubelet-config\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703886    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af7f0b87-ae82-4e3f-b10f-c85648bde428-cni-binary-copy\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703894    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82e3f885-f60e-4039-ac7d-3318d0fb27ca-dbus\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703918    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-lib-modules\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703941    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-cni-netd\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703963    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-sysctl-conf\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.703984    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-slash\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.705145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704006    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jtms\" (UniqueName: \"kubernetes.io/projected/301f05ca-4281-49fc-8f5c-e059aab4a6ce-kube-api-access-5jtms\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704028    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb780329-b558-4913-ae68-75262032b1a1-cni-binary-copy\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704059    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704071    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af7f0b87-ae82-4e3f-b10f-c85648bde428-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704082    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-env-overrides\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704119    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-multus-conf-dir\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704147    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-modprobe-d\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704174    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-ovn-node-metrics-cert\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704218    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-multus-conf-dir\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704246    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-socket-dir\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704283    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704305    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-node-log\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704329    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-ovnkube-config\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704350    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-ovnkube-script-lib\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704374    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-sys-fs\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704413    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c55g4\" (UniqueName: \"kubernetes.io/projected/1617d1a8-1f68-4475-88f9-f6fdb91a07db-kube-api-access-c55g4\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:25.705801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704439    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb780329-b558-4913-ae68-75262032b1a1-cni-binary-copy\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.706583 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704479    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-var-lib-kubelet\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.706583 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704523    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-var-lib-cni-bin\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.706583 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.704575    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb780329-b558-4913-ae68-75262032b1a1-host-var-lib-cni-bin\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.707142 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.707118    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 12:03:25.711414 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.711390    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvx9t\" (UniqueName: \"kubernetes.io/projected/4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b-kube-api-access-zvx9t\") pod \"node-ca-tgspg\" (UID: \"4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b\") " pod="openshift-image-registry/node-ca-tgspg"
Sep 16 12:03:25.712295 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.712255    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq8qr\" (UniqueName: \"kubernetes.io/projected/91c3d651-2ead-43e7-ba31-acb29b24408f-kube-api-access-lq8qr\") pod \"iptables-alerter-dhdrp\" (UID: \"91c3d651-2ead-43e7-ba31-acb29b24408f\") " pod="openshift-network-operator/iptables-alerter-dhdrp"
Sep 16 12:03:25.716213 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.716192    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tm95\" (UniqueName: \"kubernetes.io/projected/cb780329-b558-4913-ae68-75262032b1a1-kube-api-access-8tm95\") pod \"multus-p8889\" (UID: \"cb780329-b558-4913-ae68-75262032b1a1\") " pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.717776 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.717735    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwt7k\" (UniqueName: \"kubernetes.io/projected/af7f0b87-ae82-4e3f-b10f-c85648bde428-kube-api-access-nwt7k\") pod \"multus-additional-cni-plugins-9tqmt\" (UID: \"af7f0b87-ae82-4e3f-b10f-c85648bde428\") " pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.718236 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.718208    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c55g4\" (UniqueName: \"kubernetes.io/projected/1617d1a8-1f68-4475-88f9-f6fdb91a07db-kube-api-access-c55g4\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:25.756391 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.756309    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-98.ec2.internal" event={"ID":"7698f8b509c65d280b6ab05a4a563a4f","Type":"ContainerStarted","Data":"3accfc44ffe282916d23a12fe4ee8f10bceded70a44871eca1d4ff5abbefd89f"}
Sep 16 12:03:25.757726 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.757698    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal" event={"ID":"9288b06b268d24887bc1ba15eed675a9","Type":"ContainerStarted","Data":"66bf8b0292a304fd0468f8c45c84270d7d98497c7afcdb1248ff8b536557cf6c"}
Sep 16 12:03:25.805776 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805693    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-device-dir\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.805776 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805728    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82e3f885-f60e-4039-ac7d-3318d0fb27ca-kubelet-config\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:25.805776 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805743    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82e3f885-f60e-4039-ac7d-3318d0fb27ca-dbus\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:25.805776 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805761    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-lib-modules\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805784    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-cni-netd\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805798    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-sysctl-conf\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805813    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-slash\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805831    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jtms\" (UniqueName: \"kubernetes.io/projected/301f05ca-4281-49fc-8f5c-e059aab4a6ce-kube-api-access-5jtms\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805847    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805844    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-device-dir\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805862    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-env-overrides\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805877    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-modprobe-d\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805891    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-ovn-node-metrics-cert\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805906    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-socket-dir\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805921    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805935    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-cni-netd\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805963    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-node-log\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805960    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-lib-modules\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805935    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-node-log\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805888    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82e3f885-f60e-4039-ac7d-3318d0fb27ca-kubelet-config\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:25.806082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806021    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-sysctl-conf\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806037    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-modprobe-d\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805852    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82e3f885-f60e-4039-ac7d-3318d0fb27ca-dbus\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806089    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-socket-dir\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806125    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.805904    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-slash\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806192    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-ovnkube-config\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-ovnkube-script-lib\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806252    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-sys-fs\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806279    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-var-lib-kubelet\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806305    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-kubernetes\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806331    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/678bc08c-e65a-4d2f-8de1-5b73269448c2-hosts-file\") pod \"node-resolver-fk9r5\" (UID: \"678bc08c-e65a-4d2f-8de1-5b73269448c2\") " pod="openshift-dns/node-resolver-fk9r5"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-run-netns\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-sysconfig\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806394    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-sys-fs\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806403    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-sys\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.806888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806438    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/678bc08c-e65a-4d2f-8de1-5b73269448c2-hosts-file\") pod \"node-resolver-fk9r5\" (UID: \"678bc08c-e65a-4d2f-8de1-5b73269448c2\") " pod="openshift-dns/node-resolver-fk9r5"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806452    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-sys\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806440    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29zz\" (UniqueName: \"kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz\") pod \"network-check-target-bzbgg\" (UID: \"cca7ed59-7144-44f8-8421-497cb96f2aae\") " pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806484    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-var-lib-kubelet\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806493    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-systemd\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806517    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-kubelet\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-kubernetes\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806533    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-sysconfig\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806395    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-env-overrides\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806541    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-systemd-units\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806590    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-systemd-units\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806592    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-run-systemd\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806494    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-run-netns\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806602    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-systemd\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806646    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-run-systemd\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806647    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-kubelet\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806675    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-etc-openvswitch\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806698    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-log-socket\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.807751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806721    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-cni-bin\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806747    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-sysctl-d\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806749    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-ovnkube-config\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806767    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-etc-openvswitch\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806770    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-ovnkube-script-lib\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806770    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-tuned\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806822    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678bc08c-e65a-4d2f-8de1-5b73269448c2-tmp-dir\") pod \"node-resolver-fk9r5\" (UID: \"678bc08c-e65a-4d2f-8de1-5b73269448c2\") " pod="openshift-dns/node-resolver-fk9r5"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806848    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-run-openvswitch\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-cni-bin\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806865    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-sysctl-d\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-run-ovn\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806825    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-log-socket\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806894    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-run-ovn-kubernetes\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806929    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-run\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806933    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-run-openvswitch\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806950    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a2351a4-292f-4ccd-aefc-d3467485cd0e-tmp\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.806971    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-run-ovn\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807010    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzfdn\" (UniqueName: \"kubernetes.io/projected/2a2351a4-292f-4ccd-aefc-d3467485cd0e-kube-api-access-rzfdn\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.808576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807059    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kfx7\" (UniqueName: \"kubernetes.io/projected/678bc08c-e65a-4d2f-8de1-5b73269448c2-kube-api-access-7kfx7\") pod \"node-resolver-fk9r5\" (UID: \"678bc08c-e65a-4d2f-8de1-5b73269448c2\") " pod="openshift-dns/node-resolver-fk9r5"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807073    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-host-run-ovn-kubernetes\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807082    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-var-lib-openvswitch\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807114    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccq9z\" (UniqueName: \"kubernetes.io/projected/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-kube-api-access-ccq9z\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807141    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc-agent-certs\") pod \"konnectivity-agent-hpmjr\" (UID: \"b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc\") " pod="kube-system/konnectivity-agent-hpmjr"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807116    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-var-lib-openvswitch\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807165    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc-konnectivity-ca\") pod \"konnectivity-agent-hpmjr\" (UID: \"b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc\") " pod="kube-system/konnectivity-agent-hpmjr"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807188    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807213    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-registration-dir\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807238    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-host\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807333    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-host\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807339    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a2351a4-292f-4ccd-aefc-d3467485cd0e-run\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807140    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678bc08c-e65a-4d2f-8de1-5b73269448c2-tmp-dir\") pod \"node-resolver-fk9r5\" (UID: \"678bc08c-e65a-4d2f-8de1-5b73269448c2\") " pod="openshift-dns/node-resolver-fk9r5"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807397    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-registration-dir\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:25.807452    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:25.807507    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret podName:82e3f885-f60e-4039-ac7d-3318d0fb27ca nodeName:}" failed. No retries permitted until 2026-09-16 12:03:26.307491176 +0000 UTC m=+3.061449899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret") pod "global-pull-secret-syncer-xc9vl" (UID: "82e3f885-f60e-4039-ac7d-3318d0fb27ca") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:25.809422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807522    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/301f05ca-4281-49fc-8f5c-e059aab4a6ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.810236 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.807929    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc-konnectivity-ca\") pod \"konnectivity-agent-hpmjr\" (UID: \"b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc\") " pod="kube-system/konnectivity-agent-hpmjr"
Sep 16 12:03:25.810236 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.808738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-ovn-node-metrics-cert\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.810236 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.809076    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a2351a4-292f-4ccd-aefc-d3467485cd0e-etc-tuned\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.810236 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.809786    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a2351a4-292f-4ccd-aefc-d3467485cd0e-tmp\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.810236 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.810102    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc-agent-certs\") pod \"konnectivity-agent-hpmjr\" (UID: \"b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc\") " pod="kube-system/konnectivity-agent-hpmjr"
Sep 16 12:03:25.812127 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:25.812093    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:25.812127 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:25.812128    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:25.812313 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:25.812142    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d29zz for pod openshift-network-diagnostics/network-check-target-bzbgg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:25.812313 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:25.812202    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz podName:cca7ed59-7144-44f8-8421-497cb96f2aae nodeName:}" failed. No retries permitted until 2026-09-16 12:03:26.312184413 +0000 UTC m=+3.066143123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d29zz" (UniqueName: "kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz") pod "network-check-target-bzbgg" (UID: "cca7ed59-7144-44f8-8421-497cb96f2aae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:25.814200 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.814162    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jtms\" (UniqueName: \"kubernetes.io/projected/301f05ca-4281-49fc-8f5c-e059aab4a6ce-kube-api-access-5jtms\") pod \"aws-ebs-csi-driver-node-7sbdv\" (UID: \"301f05ca-4281-49fc-8f5c-e059aab4a6ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.814725 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.814694    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kfx7\" (UniqueName: \"kubernetes.io/projected/678bc08c-e65a-4d2f-8de1-5b73269448c2-kube-api-access-7kfx7\") pod \"node-resolver-fk9r5\" (UID: \"678bc08c-e65a-4d2f-8de1-5b73269448c2\") " pod="openshift-dns/node-resolver-fk9r5"
Sep 16 12:03:25.814725 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.814696    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccq9z\" (UniqueName: \"kubernetes.io/projected/ccdcfa5c-5b27-4543-936b-b2d7fc1c777d-kube-api-access-ccq9z\") pod \"ovnkube-node-xflq6\" (UID: \"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d\") " pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.814725 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.814724    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzfdn\" (UniqueName: \"kubernetes.io/projected/2a2351a4-292f-4ccd-aefc-d3467485cd0e-kube-api-access-rzfdn\") pod \"tuned-2wjkt\" (UID: \"2a2351a4-292f-4ccd-aefc-d3467485cd0e\") " pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.889780 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.889743    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tgspg"
Sep 16 12:03:25.905059 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.905015    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9tqmt"
Sep 16 12:03:25.911970 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.911945    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p8889"
Sep 16 12:03:25.919919 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.919883    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dhdrp"
Sep 16 12:03:25.928747 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.928720    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hpmjr"
Sep 16 12:03:25.937675 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.937647    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2wjkt"
Sep 16 12:03:25.946496 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.946468    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:25.954324 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.954290    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv"
Sep 16 12:03:25.962165 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:25.962139    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fk9r5"
Sep 16 12:03:26.209935 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:26.209899    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:26.210108 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:26.210039    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:26.210108 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:26.210106    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs podName:1617d1a8-1f68-4475-88f9-f6fdb91a07db nodeName:}" failed. No retries permitted until 2026-09-16 12:03:27.210088584 +0000 UTC m=+3.964047303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs") pod "network-metrics-daemon-pm6d7" (UID: "1617d1a8-1f68-4475-88f9-f6fdb91a07db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:26.310454 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:26.310420    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:26.310618 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:26.310540    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:26.310618 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:26.310609    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret podName:82e3f885-f60e-4039-ac7d-3318d0fb27ca nodeName:}" failed. No retries permitted until 2026-09-16 12:03:27.310594759 +0000 UTC m=+4.064553469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret") pod "global-pull-secret-syncer-xc9vl" (UID: "82e3f885-f60e-4039-ac7d-3318d0fb27ca") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:26.411781 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:26.411737    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29zz\" (UniqueName: \"kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz\") pod \"network-check-target-bzbgg\" (UID: \"cca7ed59-7144-44f8-8421-497cb96f2aae\") " pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:26.411956 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:26.411930    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:26.411956 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:26.411954    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:26.412054 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:26.411970    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d29zz for pod openshift-network-diagnostics/network-check-target-bzbgg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:26.412054 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:26.412044    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz podName:cca7ed59-7144-44f8-8421-497cb96f2aae nodeName:}" failed. No retries permitted until 2026-09-16 12:03:27.412023134 +0000 UTC m=+4.165981860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d29zz" (UniqueName: "kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz") pod "network-check-target-bzbgg" (UID: "cca7ed59-7144-44f8-8421-497cb96f2aae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:26.496560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:26.496473    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:26.637120 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:26.637080    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:58:25 +0000 UTC" deadline="2028-04-20 06:53:52.614596266 +0000 UTC"
Sep 16 12:03:26.637120 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:26.637112    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13962h50m25.97748738s"
Sep 16 12:03:26.876700 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:26.876164    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7f0b87_ae82_4e3f_b10f_c85648bde428.slice/crio-bd3cdad338bf6b66dc7fade4541bdbe269e6dd8bb2febeb1bbbebaf814480ef7 WatchSource:0}: Error finding container bd3cdad338bf6b66dc7fade4541bdbe269e6dd8bb2febeb1bbbebaf814480ef7: Status 404 returned error can't find the container with id bd3cdad338bf6b66dc7fade4541bdbe269e6dd8bb2febeb1bbbebaf814480ef7
Sep 16 12:03:26.881274 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:26.881148    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb780329_b558_4913_ae68_75262032b1a1.slice/crio-3836850a876514ab3c5dedd99bbbc50575a579b1e75753bbecffae4296110b21 WatchSource:0}: Error finding container 3836850a876514ab3c5dedd99bbbc50575a579b1e75753bbecffae4296110b21: Status 404 returned error can't find the container with id 3836850a876514ab3c5dedd99bbbc50575a579b1e75753bbecffae4296110b21
Sep 16 12:03:26.882098 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:26.881909    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301f05ca_4281_49fc_8f5c_e059aab4a6ce.slice/crio-c2466fbc9d7464b162f39d5ed4b31e6df05bd1745d84385b24f6ec6b8031012e WatchSource:0}: Error finding container c2466fbc9d7464b162f39d5ed4b31e6df05bd1745d84385b24f6ec6b8031012e: Status 404 returned error can't find the container with id c2466fbc9d7464b162f39d5ed4b31e6df05bd1745d84385b24f6ec6b8031012e
Sep 16 12:03:26.882991 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:26.882955    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678bc08c_e65a_4d2f_8de1_5b73269448c2.slice/crio-8c80ff9ccaf615e0937ec2ceba54a1481369e75df49a22ebec15c5a70c71f292 WatchSource:0}: Error finding container 8c80ff9ccaf615e0937ec2ceba54a1481369e75df49a22ebec15c5a70c71f292: Status 404 returned error can't find the container with id 8c80ff9ccaf615e0937ec2ceba54a1481369e75df49a22ebec15c5a70c71f292
Sep 16 12:03:26.883935 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:26.883617    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2351a4_292f_4ccd_aefc_d3467485cd0e.slice/crio-6c3f9c53e4f9f11da82ef2a3e32f2883ed265feb0afee171afbe117deb697e98 WatchSource:0}: Error finding container 6c3f9c53e4f9f11da82ef2a3e32f2883ed265feb0afee171afbe117deb697e98: Status 404 returned error can't find the container with id 6c3f9c53e4f9f11da82ef2a3e32f2883ed265feb0afee171afbe117deb697e98
Sep 16 12:03:26.884723 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:26.884702    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccdcfa5c_5b27_4543_936b_b2d7fc1c777d.slice/crio-4d919db8abbf2dd69cdb01053197865baf28d3a4b0fd0b0e20d3a81aee7143ad WatchSource:0}: Error finding container 4d919db8abbf2dd69cdb01053197865baf28d3a4b0fd0b0e20d3a81aee7143ad: Status 404 returned error can't find the container with id 4d919db8abbf2dd69cdb01053197865baf28d3a4b0fd0b0e20d3a81aee7143ad
Sep 16 12:03:26.886475 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:26.886455    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0a1d8d_4b3a_499b_ab3e_4426fcb5948b.slice/crio-09c936d74d6b89928d30147e6fd38f667d606b4e806544cd20e6beb4a56f421a WatchSource:0}: Error finding container 09c936d74d6b89928d30147e6fd38f667d606b4e806544cd20e6beb4a56f421a: Status 404 returned error can't find the container with id 09c936d74d6b89928d30147e6fd38f667d606b4e806544cd20e6beb4a56f421a
Sep 16 12:03:27.217029 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.216849    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:27.217177 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:27.216987    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:27.217177 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:27.217094    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs podName:1617d1a8-1f68-4475-88f9-f6fdb91a07db nodeName:}" failed. No retries permitted until 2026-09-16 12:03:29.217079934 +0000 UTC m=+5.971038642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs") pod "network-metrics-daemon-pm6d7" (UID: "1617d1a8-1f68-4475-88f9-f6fdb91a07db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:27.317566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.317517    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:27.317757 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:27.317738    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:27.317822 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:27.317805    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret podName:82e3f885-f60e-4039-ac7d-3318d0fb27ca nodeName:}" failed. No retries permitted until 2026-09-16 12:03:29.317787293 +0000 UTC m=+6.071746015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret") pod "global-pull-secret-syncer-xc9vl" (UID: "82e3f885-f60e-4039-ac7d-3318d0fb27ca") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:27.418624 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.418526    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29zz\" (UniqueName: \"kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz\") pod \"network-check-target-bzbgg\" (UID: \"cca7ed59-7144-44f8-8421-497cb96f2aae\") " pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:27.418833 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:27.418754    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:27.418833 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:27.418773    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:27.418833 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:27.418786    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d29zz for pod openshift-network-diagnostics/network-check-target-bzbgg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:27.419012 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:27.418845    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz podName:cca7ed59-7144-44f8-8421-497cb96f2aae nodeName:}" failed. No retries permitted until 2026-09-16 12:03:29.418827483 +0000 UTC m=+6.172786214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d29zz" (UniqueName: "kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz") pod "network-check-target-bzbgg" (UID: "cca7ed59-7144-44f8-8421-497cb96f2aae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:27.637718 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.637651    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:58:25 +0000 UTC" deadline="2028-03-04 21:35:23.945144489 +0000 UTC"
Sep 16 12:03:27.637718 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.637694    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12849h31m56.307456505s"
Sep 16 12:03:27.758616 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.753789    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:27.758616 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:27.754025    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:27.758616 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.754645    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:27.758616 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:27.754799    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:27.758616 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.754885    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:27.758616 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:27.755153    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:27.789533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.789436    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p8889" event={"ID":"cb780329-b558-4913-ae68-75262032b1a1","Type":"ContainerStarted","Data":"3836850a876514ab3c5dedd99bbbc50575a579b1e75753bbecffae4296110b21"}
Sep 16 12:03:27.804224 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.804176    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hpmjr" event={"ID":"b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc","Type":"ContainerStarted","Data":"0150a4a2728b8abf32b0b3798da89fcd18645457cb0cbb0511d5fdd7d2270114"}
Sep 16 12:03:27.810532 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.810493    2558 generic.go:356] "Generic (PLEG): container finished" podID="9288b06b268d24887bc1ba15eed675a9" containerID="09ff3644144c8c4c6c6dd96c33d1bb97505273dc2c36b5954aeb12a83aa9880f" exitCode=0
Sep 16 12:03:27.810733 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.810607    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal" event={"ID":"9288b06b268d24887bc1ba15eed675a9","Type":"ContainerDied","Data":"09ff3644144c8c4c6c6dd96c33d1bb97505273dc2c36b5954aeb12a83aa9880f"}
Sep 16 12:03:27.829029 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.828997    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-98.ec2.internal" event={"ID":"7698f8b509c65d280b6ab05a4a563a4f","Type":"ContainerStarted","Data":"4b8be0459f4d48d1f3d625427b4f8e6c43392fe6081feb65983d8935111ebf24"}
Sep 16 12:03:27.836912 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.836880    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tgspg" event={"ID":"4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b","Type":"ContainerStarted","Data":"09c936d74d6b89928d30147e6fd38f667d606b4e806544cd20e6beb4a56f421a"}
Sep 16 12:03:27.861483 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.859930    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2wjkt" event={"ID":"2a2351a4-292f-4ccd-aefc-d3467485cd0e","Type":"ContainerStarted","Data":"6c3f9c53e4f9f11da82ef2a3e32f2883ed265feb0afee171afbe117deb697e98"}
Sep 16 12:03:27.861483 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.861075    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-98.ec2.internal" podStartSLOduration=3.861055842 podStartE2EDuration="3.861055842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:27.860254939 +0000 UTC m=+4.614213671" watchObservedRunningTime="2026-09-16 12:03:27.861055842 +0000 UTC m=+4.615014576"
Sep 16 12:03:27.883082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.881097    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fk9r5" event={"ID":"678bc08c-e65a-4d2f-8de1-5b73269448c2","Type":"ContainerStarted","Data":"8c80ff9ccaf615e0937ec2ceba54a1481369e75df49a22ebec15c5a70c71f292"}
Sep 16 12:03:27.890281 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.890168    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv" event={"ID":"301f05ca-4281-49fc-8f5c-e059aab4a6ce","Type":"ContainerStarted","Data":"c2466fbc9d7464b162f39d5ed4b31e6df05bd1745d84385b24f6ec6b8031012e"}
Sep 16 12:03:27.897334 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.897300    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tqmt" event={"ID":"af7f0b87-ae82-4e3f-b10f-c85648bde428","Type":"ContainerStarted","Data":"bd3cdad338bf6b66dc7fade4541bdbe269e6dd8bb2febeb1bbbebaf814480ef7"}
Sep 16 12:03:27.905664 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.905630    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dhdrp" event={"ID":"91c3d651-2ead-43e7-ba31-acb29b24408f","Type":"ContainerStarted","Data":"748046e4c210f94f4f501725d31e95a1835572d4f4964283320ab3d08b3c34ad"}
Sep 16 12:03:27.915369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:27.915331    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6" event={"ID":"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d","Type":"ContainerStarted","Data":"4d919db8abbf2dd69cdb01053197865baf28d3a4b0fd0b0e20d3a81aee7143ad"}
Sep 16 12:03:28.926866 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:28.926604    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal" event={"ID":"9288b06b268d24887bc1ba15eed675a9","Type":"ContainerStarted","Data":"b76c9d9c43ee6dea9e1841df599e37af29be35070b0f692acfbcc296b58ced31"}
Sep 16 12:03:29.236567 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:29.236459    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:29.236752 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:29.236633    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:29.236752 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:29.236704    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs podName:1617d1a8-1f68-4475-88f9-f6fdb91a07db nodeName:}" failed. No retries permitted until 2026-09-16 12:03:33.236685878 +0000 UTC m=+9.990644594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs") pod "network-metrics-daemon-pm6d7" (UID: "1617d1a8-1f68-4475-88f9-f6fdb91a07db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:29.337139 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:29.337103    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:29.337298 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:29.337241    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:29.337354 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:29.337302    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret podName:82e3f885-f60e-4039-ac7d-3318d0fb27ca nodeName:}" failed. No retries permitted until 2026-09-16 12:03:33.337284061 +0000 UTC m=+10.091242779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret") pod "global-pull-secret-syncer-xc9vl" (UID: "82e3f885-f60e-4039-ac7d-3318d0fb27ca") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:29.438135 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:29.438038    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29zz\" (UniqueName: \"kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz\") pod \"network-check-target-bzbgg\" (UID: \"cca7ed59-7144-44f8-8421-497cb96f2aae\") " pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:29.438296 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:29.438239    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:29.438296 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:29.438258    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:29.438296 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:29.438270    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d29zz for pod openshift-network-diagnostics/network-check-target-bzbgg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:29.438461 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:29.438324    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz podName:cca7ed59-7144-44f8-8421-497cb96f2aae nodeName:}" failed. No retries permitted until 2026-09-16 12:03:33.438306106 +0000 UTC m=+10.192264820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d29zz" (UniqueName: "kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz") pod "network-check-target-bzbgg" (UID: "cca7ed59-7144-44f8-8421-497cb96f2aae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:29.752744 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:29.752712    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:29.752912 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:29.752846    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:29.753240 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:29.753223    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:29.753340 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:29.753321    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:29.753456 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:29.753443    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:29.753575 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:29.753540    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:31.752456 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:31.751980    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:31.752456 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:31.752114    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:31.753025 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:31.752468    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:31.753025 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:31.752605    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:31.753025 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:31.752655    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:31.753025 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:31.752750    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:33.270225 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:33.270183    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:33.270710 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:33.270335    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:33.270710 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:33.270404    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs podName:1617d1a8-1f68-4475-88f9-f6fdb91a07db nodeName:}" failed. No retries permitted until 2026-09-16 12:03:41.270384607 +0000 UTC m=+18.024343335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs") pod "network-metrics-daemon-pm6d7" (UID: "1617d1a8-1f68-4475-88f9-f6fdb91a07db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:33.371253 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:33.371214    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:33.371446 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:33.371385    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:33.372371 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:33.371464    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret podName:82e3f885-f60e-4039-ac7d-3318d0fb27ca nodeName:}" failed. No retries permitted until 2026-09-16 12:03:41.371443891 +0000 UTC m=+18.125402608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret") pod "global-pull-secret-syncer-xc9vl" (UID: "82e3f885-f60e-4039-ac7d-3318d0fb27ca") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:33.472823 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:33.472707    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29zz\" (UniqueName: \"kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz\") pod \"network-check-target-bzbgg\" (UID: \"cca7ed59-7144-44f8-8421-497cb96f2aae\") " pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:33.473023 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:33.472905    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:33.473023 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:33.472927    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:33.473023 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:33.472941    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d29zz for pod openshift-network-diagnostics/network-check-target-bzbgg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:33.473023 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:33.473005    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz podName:cca7ed59-7144-44f8-8421-497cb96f2aae nodeName:}" failed. No retries permitted until 2026-09-16 12:03:41.472986184 +0000 UTC m=+18.226944906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d29zz" (UniqueName: "kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz") pod "network-check-target-bzbgg" (UID: "cca7ed59-7144-44f8-8421-497cb96f2aae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:33.753381 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:33.752458    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:33.753381 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:33.752559    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:33.753381 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:33.752582    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:33.753381 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:33.752692    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:33.753381 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:33.752749    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:33.753381 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:33.752819    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:35.751672 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:35.751634    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:35.752098 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:35.751764    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:35.752098 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:35.751773    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:35.752098 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:35.751805    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:35.752098 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:35.751873    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:35.752098 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:35.751936    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:37.752472 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:37.752438    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:37.752879 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:37.752440    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:37.752879 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:37.752575    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:37.752879 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:37.752594    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:37.752879 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:37.752701    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:37.752879 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:37.752778    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:39.751729 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:39.751692    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:39.751729 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:39.751729    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:39.752208 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:39.751696    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:39.752208 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:39.751828    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:39.752208 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:39.751904    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:39.752208 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:39.752001    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:41.330289 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:41.330248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:41.330789 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:41.330411    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:41.330789 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:41.330492    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs podName:1617d1a8-1f68-4475-88f9-f6fdb91a07db nodeName:}" failed. No retries permitted until 2026-09-16 12:03:57.33047417 +0000 UTC m=+34.084432889 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs") pod "network-metrics-daemon-pm6d7" (UID: "1617d1a8-1f68-4475-88f9-f6fdb91a07db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:41.431283 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:41.431248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:41.431466 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:41.431368    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:41.431466 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:41.431433    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret podName:82e3f885-f60e-4039-ac7d-3318d0fb27ca nodeName:}" failed. No retries permitted until 2026-09-16 12:03:57.431416649 +0000 UTC m=+34.185375357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret") pod "global-pull-secret-syncer-xc9vl" (UID: "82e3f885-f60e-4039-ac7d-3318d0fb27ca") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:41.532283 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:41.532244    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29zz\" (UniqueName: \"kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz\") pod \"network-check-target-bzbgg\" (UID: \"cca7ed59-7144-44f8-8421-497cb96f2aae\") " pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:41.532448 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:41.532375    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:41.532448 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:41.532394    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:41.532448 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:41.532404    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d29zz for pod openshift-network-diagnostics/network-check-target-bzbgg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:41.532644 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:41.532454    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz podName:cca7ed59-7144-44f8-8421-497cb96f2aae nodeName:}" failed. No retries permitted until 2026-09-16 12:03:57.532439799 +0000 UTC m=+34.286398518 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d29zz" (UniqueName: "kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz") pod "network-check-target-bzbgg" (UID: "cca7ed59-7144-44f8-8421-497cb96f2aae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:41.751817 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:41.751773    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:41.752001 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:41.751780    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:41.752001 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:41.751902    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:41.752001 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:41.751918    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:41.752147 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:41.752037    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:41.752147 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:41.752116    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:43.753065 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:43.752852    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:43.753524 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:43.752929    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:43.753524 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:43.753157    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:43.753524 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:43.753259    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:43.753524 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:43.752945    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:43.753524 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:43.753361    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:44.958446 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.958195    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tgspg" event={"ID":"4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b","Type":"ContainerStarted","Data":"fbdf5edb9710e66d1449fc8bd98845e57ccc57a85aa329fa51414ad741436eaa"}
Sep 16 12:03:44.959853 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.959826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2wjkt" event={"ID":"2a2351a4-292f-4ccd-aefc-d3467485cd0e","Type":"ContainerStarted","Data":"ad3895de84f3f09dce593e62e6069942a2d51909106af6fa7548d104aed6697f"}
Sep 16 12:03:44.961572 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.961527    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fk9r5" event={"ID":"678bc08c-e65a-4d2f-8de1-5b73269448c2","Type":"ContainerStarted","Data":"82c955167bd38d8b4ba0bc21ea32e372a77adcf98e4b39e59030e4934281e2cd"}
Sep 16 12:03:44.963423 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.963396    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv" event={"ID":"301f05ca-4281-49fc-8f5c-e059aab4a6ce","Type":"ContainerStarted","Data":"ac1bd6bf0a613704b7cd5a222b9066e78c85422bd63f5d2beb8d451d9a4bc0ca"}
Sep 16 12:03:44.965215 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.965190    2558 generic.go:356] "Generic (PLEG): container finished" podID="af7f0b87-ae82-4e3f-b10f-c85648bde428" containerID="6db930474201560241af06f9576b4932890e629e5747552f47c00368368cc393" exitCode=0
Sep 16 12:03:44.965308 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.965270    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tqmt" event={"ID":"af7f0b87-ae82-4e3f-b10f-c85648bde428","Type":"ContainerDied","Data":"6db930474201560241af06f9576b4932890e629e5747552f47c00368368cc393"}
Sep 16 12:03:44.967670 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.967648    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6" event={"ID":"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d","Type":"ContainerStarted","Data":"4461e7c99279456175b2e2a9e8e7312488c48d17184d039328349c4f2863e55b"}
Sep 16 12:03:44.967755 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.967718    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6" event={"ID":"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d","Type":"ContainerStarted","Data":"36a426cb0c05a564142c94e3bb86965a87b9d87ef96c2fd57e93018a98ed9089"}
Sep 16 12:03:44.967819 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.967768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6" event={"ID":"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d","Type":"ContainerStarted","Data":"705d5045c489556a6a9e55fd0d4fd649539613d3fc58685267d5e191a01ff02a"}
Sep 16 12:03:44.969259 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.969236    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p8889" event={"ID":"cb780329-b558-4913-ae68-75262032b1a1","Type":"ContainerStarted","Data":"def4867c5d48212c5e0ce934b701e37b36946b857ade0254dfd0c27c086b1e8c"}
Sep 16 12:03:44.970579 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.970542    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hpmjr" event={"ID":"b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc","Type":"ContainerStarted","Data":"c133c44cdbfe915514e456e37fe14228ce43a026acd503c892b65bdb01ad2033"}
Sep 16 12:03:44.972566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.972498    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-98.ec2.internal" podStartSLOduration=20.97248018 podStartE2EDuration="20.97248018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:28.947764778 +0000 UTC m=+5.701723511" watchObservedRunningTime="2026-09-16 12:03:44.97248018 +0000 UTC m=+21.726438929"
Sep 16 12:03:44.985990 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.985953    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hpmjr" podStartSLOduration=10.664440583 podStartE2EDuration="20.985938071s" totalImagesPullingTime="10.321497488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:44.985655199 +0000 UTC m=+21.739613911" watchObservedRunningTime="2026-09-16 12:03:44.985938071 +0000 UTC m=+21.739896804"
Sep 16 12:03:44.986351 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:44.986331    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tgspg" podStartSLOduration=3.297703774 podStartE2EDuration="20.986325971s" totalImagesPullingTime="17.688622197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:44.972006829 +0000 UTC m=+21.725965565" watchObservedRunningTime="2026-09-16 12:03:44.986325971 +0000 UTC m=+21.740284700"
Sep 16 12:03:45.005823 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:45.005776    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p8889" podStartSLOduration=3.195643904 podStartE2EDuration="21.005760994s" totalImagesPullingTime="17.81011709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:45.005674114 +0000 UTC m=+21.759632859" watchObservedRunningTime="2026-09-16 12:03:45.005760994 +0000 UTC m=+21.759719725"
Sep 16 12:03:45.021278 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:45.021238    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2wjkt" podStartSLOduration=3.328310775 podStartE2EDuration="21.021224256s" totalImagesPullingTime="17.692913481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:45.020997444 +0000 UTC m=+21.774956175" watchObservedRunningTime="2026-09-16 12:03:45.021224256 +0000 UTC m=+21.775182987"
Sep 16 12:03:45.035197 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:45.035158    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fk9r5" podStartSLOduration=3.3438635740000002 podStartE2EDuration="21.035143764s" totalImagesPullingTime="17.69128019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:45.034975222 +0000 UTC m=+21.788933953" watchObservedRunningTime="2026-09-16 12:03:45.035143764 +0000 UTC m=+21.789102494"
Sep 16 12:03:45.751607 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:45.751574    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:45.751796 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:45.751742    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:45.751796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:45.751579    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:45.751915 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:45.751829    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:45.751915 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:45.751574    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:45.752016 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:45.751915    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:45.974854 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:45.974818    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dhdrp" event={"ID":"91c3d651-2ead-43e7-ba31-acb29b24408f","Type":"ContainerStarted","Data":"8b68844c3e88238d441ed65cf2ca57dc8e6838092c538337f739238b2f86cfc3"}
Sep 16 12:03:45.979675 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:45.979645    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6" event={"ID":"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d","Type":"ContainerStarted","Data":"e2030ca0ace038f131b8dc8e60fca1a8b390f35ad95100d8e6c25ce198ea65c7"}
Sep 16 12:03:45.979804 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:45.979726    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6" event={"ID":"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d","Type":"ContainerStarted","Data":"5776c747dcd2e1aa2e13720dd5af32400a9ec0c561edf80102624910434a07eb"}
Sep 16 12:03:45.979804 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:45.979779    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6" event={"ID":"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d","Type":"ContainerStarted","Data":"5f165cd4c152170e48680e7a716884a6a5124be1b6e664eb977c244a526aa8d4"}
Sep 16 12:03:46.347162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:46.347135    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 12:03:46.794797 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:46.794675    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:03:46.347156741Z","UUID":"639d8745-c75e-4d91-b13f-15203ae7f640","Handler":null,"Name":"","Endpoint":""}
Sep 16 12:03:46.796362 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:46.796338    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 12:03:46.796488 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:46.796371    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 12:03:46.984621 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:46.984581    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv" event={"ID":"301f05ca-4281-49fc-8f5c-e059aab4a6ce","Type":"ContainerStarted","Data":"b65c1c7cb582fe68b2da04de401c7210caff4aac95420297666489f22ca427e6"}
Sep 16 12:03:46.999400 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:46.999352    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dhdrp" podStartSLOduration=5.315374784 podStartE2EDuration="22.999316131s" totalImagesPullingTime="17.683941347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:46.999267309 +0000 UTC m=+23.753226041" watchObservedRunningTime="2026-09-16 12:03:46.999316131 +0000 UTC m=+23.753274864"
Sep 16 12:03:47.751768 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:47.751509    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:47.751955 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:47.751524    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:47.751955 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:47.751868    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:47.751955 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:47.751935    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:47.751955 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:47.751601    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:47.752157 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:47.752013    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:47.988400 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:47.988366    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv" event={"ID":"301f05ca-4281-49fc-8f5c-e059aab4a6ce","Type":"ContainerStarted","Data":"68413805db4dbed8b232985145e1162bedeae15d26ed8f47768bce1bad8bc987"}
Sep 16 12:03:47.991717 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:47.991691    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6" event={"ID":"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d","Type":"ContainerStarted","Data":"516ad487fd975d409a644d0a5308a994c99a0e403269b758f9ff8c359d0d2604"}
Sep 16 12:03:48.007435 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:48.007367    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbdv" podStartSLOduration=3.919226167 podStartE2EDuration="24.007353212s" totalImagesPullingTime="20.088127045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:48.00627162 +0000 UTC m=+24.760230350" watchObservedRunningTime="2026-09-16 12:03:48.007353212 +0000 UTC m=+24.761311942"
Sep 16 12:03:49.537533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:49.537497    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hpmjr"
Sep 16 12:03:49.538021 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:49.538004    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hpmjr"
Sep 16 12:03:49.752370 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:49.752208    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:49.752504 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:49.752208    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:49.752504 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:49.752483    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:49.752633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:49.752225    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:49.752633 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:49.752578    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:49.752783 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:49.752750    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:49.997641 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:49.997565    2558 generic.go:356] "Generic (PLEG): container finished" podID="af7f0b87-ae82-4e3f-b10f-c85648bde428" containerID="4c58ff320dd40575c81cec57f0f01700a7ad38a0a80160403ec290604fb0fc82" exitCode=0
Sep 16 12:03:49.997770 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:49.997651    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tqmt" event={"ID":"af7f0b87-ae82-4e3f-b10f-c85648bde428","Type":"ContainerDied","Data":"4c58ff320dd40575c81cec57f0f01700a7ad38a0a80160403ec290604fb0fc82"}
Sep 16 12:03:50.001556 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:50.001516    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6" event={"ID":"ccdcfa5c-5b27-4543-936b-b2d7fc1c777d","Type":"ContainerStarted","Data":"fd47898960ba9ffa420f1e88dec09f0e2cc2b149e0fd06b042e30b804739b40b"}
Sep 16 12:03:50.001895 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:50.001877    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hpmjr"
Sep 16 12:03:50.002290 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:50.002267    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hpmjr"
Sep 16 12:03:50.054674 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:50.054631    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6" podStartSLOduration=8.347267361 podStartE2EDuration="26.0546171s" totalImagesPullingTime="17.707349739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:50.053363642 +0000 UTC m=+26.807322372" watchObservedRunningTime="2026-09-16 12:03:50.0546171 +0000 UTC m=+26.808575830"
Sep 16 12:03:51.004288 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:51.003835    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:51.004288 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:51.003924    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:51.004288 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:51.003976    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:51.019914 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:51.019889    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:51.020402 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:51.020382    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:03:51.485109 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:51.485080    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pm6d7"]
Sep 16 12:03:51.485267 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:51.485253    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:51.485369 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:51.485351    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:51.488627 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:51.488595    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bzbgg"]
Sep 16 12:03:51.488756 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:51.488710    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:51.488829 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:51.488791    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:51.489263 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:51.489237    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xc9vl"]
Sep 16 12:03:51.489377 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:51.489343    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:51.491871 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:51.489435    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:52.007421 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:52.007387    2558 generic.go:356] "Generic (PLEG): container finished" podID="af7f0b87-ae82-4e3f-b10f-c85648bde428" containerID="423d53b1d2873178c3065f663ccf488e39ca8044f86d0eb8d3db9d9afa3f297a" exitCode=0
Sep 16 12:03:52.008021 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:52.007462    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tqmt" event={"ID":"af7f0b87-ae82-4e3f-b10f-c85648bde428","Type":"ContainerDied","Data":"423d53b1d2873178c3065f663ccf488e39ca8044f86d0eb8d3db9d9afa3f297a"}
Sep 16 12:03:53.012588 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:53.012307    2558 generic.go:356] "Generic (PLEG): container finished" podID="af7f0b87-ae82-4e3f-b10f-c85648bde428" containerID="d21eb08a7149168a29895b52314f380beb3033034e7482dfbd8b1e8b67796ca8" exitCode=0
Sep 16 12:03:53.012588 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:53.012387    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tqmt" event={"ID":"af7f0b87-ae82-4e3f-b10f-c85648bde428","Type":"ContainerDied","Data":"d21eb08a7149168a29895b52314f380beb3033034e7482dfbd8b1e8b67796ca8"}
Sep 16 12:03:53.756540 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:53.756449    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:53.756540 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:53.756525    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:53.756540 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:53.756613    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:53.756540 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:53.756603    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:53.756930 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:53.756686    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:53.756930 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:53.756752    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:55.752372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:55.752340    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:55.752849 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:55.752340    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:55.752849 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:55.752453    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:55.752849 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:55.752538    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:55.752849 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:55.752340    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:55.752849 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:55.752674    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:57.346757 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:57.346721    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:57.347204 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:57.346899    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:57.347204 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:57.346977    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs podName:1617d1a8-1f68-4475-88f9-f6fdb91a07db nodeName:}" failed. No retries permitted until 2026-09-16 12:04:29.346961395 +0000 UTC m=+66.100920131 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs") pod "network-metrics-daemon-pm6d7" (UID: "1617d1a8-1f68-4475-88f9-f6fdb91a07db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:57.447799 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:57.447759    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:57.447980 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:57.447928    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:57.448052 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:57.448010    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret podName:82e3f885-f60e-4039-ac7d-3318d0fb27ca nodeName:}" failed. No retries permitted until 2026-09-16 12:04:29.447988701 +0000 UTC m=+66.201947415 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret") pod "global-pull-secret-syncer-xc9vl" (UID: "82e3f885-f60e-4039-ac7d-3318d0fb27ca") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:57.548988 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:57.548957    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29zz\" (UniqueName: \"kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz\") pod \"network-check-target-bzbgg\" (UID: \"cca7ed59-7144-44f8-8421-497cb96f2aae\") " pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:57.549150 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:57.549079    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:57.549150 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:57.549093    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:57.549150 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:57.549101    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d29zz for pod openshift-network-diagnostics/network-check-target-bzbgg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:57.549272 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:57.549159    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz podName:cca7ed59-7144-44f8-8421-497cb96f2aae nodeName:}" failed. No retries permitted until 2026-09-16 12:04:29.549140268 +0000 UTC m=+66.303098983 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d29zz" (UniqueName: "kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz") pod "network-check-target-bzbgg" (UID: "cca7ed59-7144-44f8-8421-497cb96f2aae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:57.751574 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:57.751460    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:57.751574 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:57.751460    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:57.751784 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:57.751636    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:03:57.751784 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:57.751696    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:57.751891 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:57.751797    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzbgg" podUID="cca7ed59-7144-44f8-8421-497cb96f2aae"
Sep 16 12:03:57.751891 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:57.751876    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9vl" podUID="82e3f885-f60e-4039-ac7d-3318d0fb27ca"
Sep 16 12:03:58.530077 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.530008    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-98.ec2.internal" event="NodeReady"
Sep 16 12:03:58.530463 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.530126    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 12:03:58.562880 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.562856    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f8785c58c-slgb8"]
Sep 16 12:03:58.577581 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.577524    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xz5cp"]
Sep 16 12:03:58.577666 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.577646    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.580516 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.579610    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-z95hj\""
Sep 16 12:03:58.580516 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.579917    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 12:03:58.580516 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.580105    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 12:03:58.580516 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.580124    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 12:03:58.600138 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.599985    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sshpk"]
Sep 16 12:03:58.600138 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.600017    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:03:58.600138 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.600121    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 12:03:58.602419 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.602348    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 12:03:58.602419 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.602415    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7d4qr\""
Sep 16 12:03:58.602747 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.602582    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 12:03:58.618026 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.618004    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f8785c58c-slgb8"]
Sep 16 12:03:58.618141 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.618040    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sshpk"]
Sep 16 12:03:58.618141 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.618060    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xz5cp"]
Sep 16 12:03:58.618141 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.618132    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sshpk"
Sep 16 12:03:58.620137 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.620115    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nn9m4\""
Sep 16 12:03:58.620236 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.620115    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 12:03:58.620236 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.620180    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 12:03:58.620422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.620410    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 12:03:58.758288 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758256    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwjxl\" (UniqueName: \"kubernetes.io/projected/1f341464-bbd6-4a15-82fe-51149ded0036-kube-api-access-cwjxl\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:03:58.758402 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758338    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgdwq\" (UniqueName: \"kubernetes.io/projected/3ef9d7a4-3c9d-4802-8871-18079adb56ea-kube-api-access-pgdwq\") pod \"ingress-canary-sshpk\" (UID: \"3ef9d7a4-3c9d-4802-8871-18079adb56ea\") " pod="openshift-ingress-canary/ingress-canary-sshpk"
Sep 16 12:03:58.758402 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758369    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-certificates\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.758402 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758390    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-bound-sa-token\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.758540 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758441    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7f8m\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-kube-api-access-n7f8m\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.758540 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758490    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:03:58.758540 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758535    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ef9d7a4-3c9d-4802-8871-18079adb56ea-cert\") pod \"ingress-canary-sshpk\" (UID: \"3ef9d7a4-3c9d-4802-8871-18079adb56ea\") " pod="openshift-ingress-canary/ingress-canary-sshpk"
Sep 16 12:03:58.758666 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758579    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-trusted-ca\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.758666 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758597    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f341464-bbd6-4a15-82fe-51149ded0036-tmp-dir\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:03:58.758666 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758612    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f341464-bbd6-4a15-82fe-51149ded0036-config-volume\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:03:58.758666 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758628    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.758666 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758651    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-image-registry-private-configuration\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.758820 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758686    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-ca-trust-extracted\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.758820 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.758712    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-installation-pull-secrets\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.859437 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.859398    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgdwq\" (UniqueName: \"kubernetes.io/projected/3ef9d7a4-3c9d-4802-8871-18079adb56ea-kube-api-access-pgdwq\") pod \"ingress-canary-sshpk\" (UID: \"3ef9d7a4-3c9d-4802-8871-18079adb56ea\") " pod="openshift-ingress-canary/ingress-canary-sshpk"
Sep 16 12:03:58.859437 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.859435    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-certificates\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.859763 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.859452    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-bound-sa-token\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.859763 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.859704    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7f8m\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-kube-api-access-n7f8m\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.859763 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.859742    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:03:58.859943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.859775    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ef9d7a4-3c9d-4802-8871-18079adb56ea-cert\") pod \"ingress-canary-sshpk\" (UID: \"3ef9d7a4-3c9d-4802-8871-18079adb56ea\") " pod="openshift-ingress-canary/ingress-canary-sshpk"
Sep 16 12:03:58.859943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.859800    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-trusted-ca\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.859943 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:58.859848    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:03:58.859943 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:58.859912    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls podName:1f341464-bbd6-4a15-82fe-51149ded0036 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:59.359890804 +0000 UTC m=+36.113849527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls") pod "dns-default-xz5cp" (UID: "1f341464-bbd6-4a15-82fe-51149ded0036") : secret "dns-default-metrics-tls" not found
Sep 16 12:03:58.859943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.859929    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f341464-bbd6-4a15-82fe-51149ded0036-tmp-dir\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:03:58.860230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.859962    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f341464-bbd6-4a15-82fe-51149ded0036-config-volume\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:03:58.860230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.859990    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.860230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.860022    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-image-registry-private-configuration\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.860230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.860059    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-ca-trust-extracted\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.860230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.860085    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-installation-pull-secrets\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.860230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.860150    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwjxl\" (UniqueName: \"kubernetes.io/projected/1f341464-bbd6-4a15-82fe-51149ded0036-kube-api-access-cwjxl\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:03:58.860230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.860194    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-certificates\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.860669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.860281    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f341464-bbd6-4a15-82fe-51149ded0036-tmp-dir\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:03:58.860669 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:58.860371    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:03:58.860669 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:58.860386    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8785c58c-slgb8: secret "image-registry-tls" not found
Sep 16 12:03:58.860669 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:58.860445    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls podName:6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:59.36042783 +0000 UTC m=+36.114386553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls") pod "image-registry-6f8785c58c-slgb8" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01") : secret "image-registry-tls" not found
Sep 16 12:03:58.860862 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.860712    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-ca-trust-extracted\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.860862 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.860713    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f341464-bbd6-4a15-82fe-51149ded0036-config-volume\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:03:58.860862 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.860715    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-trusted-ca\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.864068 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.864046    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-image-registry-private-configuration\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.864068 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.864055    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-installation-pull-secrets\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.864068 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.864109    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ef9d7a4-3c9d-4802-8871-18079adb56ea-cert\") pod \"ingress-canary-sshpk\" (UID: \"3ef9d7a4-3c9d-4802-8871-18079adb56ea\") " pod="openshift-ingress-canary/ingress-canary-sshpk"
Sep 16 12:03:58.867909 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.867888    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-bound-sa-token\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.868093 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.868073    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7f8m\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-kube-api-access-n7f8m\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:58.868517 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.868501    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgdwq\" (UniqueName: \"kubernetes.io/projected/3ef9d7a4-3c9d-4802-8871-18079adb56ea-kube-api-access-pgdwq\") pod \"ingress-canary-sshpk\" (UID: \"3ef9d7a4-3c9d-4802-8871-18079adb56ea\") " pod="openshift-ingress-canary/ingress-canary-sshpk"
Sep 16 12:03:58.869169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.869154    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwjxl\" (UniqueName: \"kubernetes.io/projected/1f341464-bbd6-4a15-82fe-51149ded0036-kube-api-access-cwjxl\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:03:58.938429 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:58.938390    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sshpk"
Sep 16 12:03:59.028761 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.028575    2558 generic.go:356] "Generic (PLEG): container finished" podID="af7f0b87-ae82-4e3f-b10f-c85648bde428" containerID="7d77277f46839272dda0d5df2bfbf62ea231780530be1d4dcddc282429a01e5b" exitCode=0
Sep 16 12:03:59.028907 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.028623    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tqmt" event={"ID":"af7f0b87-ae82-4e3f-b10f-c85648bde428","Type":"ContainerDied","Data":"7d77277f46839272dda0d5df2bfbf62ea231780530be1d4dcddc282429a01e5b"}
Sep 16 12:03:59.269282 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.269254    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sshpk"]
Sep 16 12:03:59.272708 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:03:59.272686    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef9d7a4_3c9d_4802_8871_18079adb56ea.slice/crio-c03c0eef53079be5e319b2449cb19549d0bfdd12f0011cfc632e0487252c12de WatchSource:0}: Error finding container c03c0eef53079be5e319b2449cb19549d0bfdd12f0011cfc632e0487252c12de: Status 404 returned error can't find the container with id c03c0eef53079be5e319b2449cb19549d0bfdd12f0011cfc632e0487252c12de
Sep 16 12:03:59.363829 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.363801    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:03:59.363978 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.363842    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:03:59.363978 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:59.363948    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:03:59.363978 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:59.363950    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:03:59.364078 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:59.364016    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls podName:1f341464-bbd6-4a15-82fe-51149ded0036 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:00.36399975 +0000 UTC m=+37.117958459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls") pod "dns-default-xz5cp" (UID: "1f341464-bbd6-4a15-82fe-51149ded0036") : secret "dns-default-metrics-tls" not found
Sep 16 12:03:59.364078 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:59.363959    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8785c58c-slgb8: secret "image-registry-tls" not found
Sep 16 12:03:59.364160 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:03:59.364081    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls podName:6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:00.364070066 +0000 UTC m=+37.118028775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls") pod "image-registry-6f8785c58c-slgb8" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01") : secret "image-registry-tls" not found
Sep 16 12:03:59.751947 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.751869    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:03:59.752501 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.751869    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:03:59.752501 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.751872    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:03:59.754943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.754771    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:03:59.754943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.754802    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:03:59.754943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.754818    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kdnz\""
Sep 16 12:03:59.754943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.754878    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gvq92\""
Sep 16 12:03:59.755192 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.754974    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:03:59.755192 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:03:59.755095    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:04:00.032418 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:00.032337    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sshpk" event={"ID":"3ef9d7a4-3c9d-4802-8871-18079adb56ea","Type":"ContainerStarted","Data":"c03c0eef53079be5e319b2449cb19549d0bfdd12f0011cfc632e0487252c12de"}
Sep 16 12:04:00.035900 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:00.035872    2558 generic.go:356] "Generic (PLEG): container finished" podID="af7f0b87-ae82-4e3f-b10f-c85648bde428" containerID="32963d59df44cc4713eb33c599d3c62e5ee388b9a6ce22278e3c2e7497e65cb3" exitCode=0
Sep 16 12:04:00.036036 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:00.035925    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tqmt" event={"ID":"af7f0b87-ae82-4e3f-b10f-c85648bde428","Type":"ContainerDied","Data":"32963d59df44cc4713eb33c599d3c62e5ee388b9a6ce22278e3c2e7497e65cb3"}
Sep 16 12:04:00.371823 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:00.371791    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:04:00.372008 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:00.371839    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:04:00.372008 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:00.371942    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:00.372008 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:00.371999    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls podName:1f341464-bbd6-4a15-82fe-51149ded0036 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:02.37198565 +0000 UTC m=+39.125944362 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls") pod "dns-default-xz5cp" (UID: "1f341464-bbd6-4a15-82fe-51149ded0036") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:00.372196 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:00.371945    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:04:00.372196 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:00.372061    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8785c58c-slgb8: secret "image-registry-tls" not found
Sep 16 12:04:00.372196 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:00.372125    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls podName:6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:02.372106693 +0000 UTC m=+39.126065408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls") pod "image-registry-6f8785c58c-slgb8" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01") : secret "image-registry-tls" not found
Sep 16 12:04:00.682912 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:00.682830    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:00.771389 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:00.771350    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:01.042344 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:01.042317    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tqmt" event={"ID":"af7f0b87-ae82-4e3f-b10f-c85648bde428","Type":"ContainerStarted","Data":"03eedb11041af2e2e7d514d19fff48f7184d3a5faec981603c1da505ba1442bd"}
Sep 16 12:04:01.064863 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:01.064792    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9tqmt" podStartSLOduration=7.492969878 podStartE2EDuration="37.064769813s" totalImagesPullingTime="29.571799935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:01.063643384 +0000 UTC m=+37.817602119" watchObservedRunningTime="2026-09-16 12:04:01.064769813 +0000 UTC m=+37.818728544"
Sep 16 12:04:01.186869 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:01.186819    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:01.240688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:01.240652    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:01.583285 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:01.583110    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:02.045791 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:02.045744    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sshpk" event={"ID":"3ef9d7a4-3c9d-4802-8871-18079adb56ea","Type":"ContainerStarted","Data":"e2fe4b606eb21d6ecbe1691173b4388f0ff265a0352ef62730086aa2ef7ba45e"}
Sep 16 12:04:02.060725 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:02.060651    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sshpk" podStartSLOduration=1.831510105 podStartE2EDuration="4.060628013s" totalImagesPullingTime="2.229117908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:02.059485431 +0000 UTC m=+38.813444164" watchObservedRunningTime="2026-09-16 12:04:02.060628013 +0000 UTC m=+38.814586743"
Sep 16 12:04:02.389100 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:02.389070    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:04:02.389295 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:02.389115    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:04:02.389295 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:02.389200    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:02.389295 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:02.389254    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls podName:1f341464-bbd6-4a15-82fe-51149ded0036 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:06.38924028 +0000 UTC m=+43.143198994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls") pod "dns-default-xz5cp" (UID: "1f341464-bbd6-4a15-82fe-51149ded0036") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:02.389295 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:02.389265    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:04:02.389295 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:02.389278    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8785c58c-slgb8: secret "image-registry-tls" not found
Sep 16 12:04:02.389501 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:02.389324    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls podName:6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:06.389308315 +0000 UTC m=+43.143267044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls") pod "image-registry-6f8785c58c-slgb8" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01") : secret "image-registry-tls" not found
Sep 16 12:04:02.758659 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:02.758582    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:02.810786 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:02.810755    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:03.199737 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:03.199684    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:03.263229 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:03.263189    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:03.504573 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:03.504476    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:03.605496 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:03.605469    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:06.066238 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:06.066206    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:06.167648 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:06.167610    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:06.268155 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:06.268118    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:06.322226 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:06.322151    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:06.417710 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:06.417673    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:04:06.417710 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:06.417715    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:04:06.417939 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:06.417829    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:04:06.417939 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:06.417843    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8785c58c-slgb8: secret "image-registry-tls" not found
Sep 16 12:04:06.417939 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:06.417827    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:06.417939 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:06.417908    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls podName:6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:14.417889073 +0000 UTC m=+51.171847783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls") pod "image-registry-6f8785c58c-slgb8" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01") : secret "image-registry-tls" not found
Sep 16 12:04:06.417939 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:06.417926    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls podName:1f341464-bbd6-4a15-82fe-51149ded0036 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:14.417918011 +0000 UTC m=+51.171876723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls") pod "dns-default-xz5cp" (UID: "1f341464-bbd6-4a15-82fe-51149ded0036") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:14.472342 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:14.472303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:04:14.472342 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:14.472344    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:04:14.472816 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:14.472438    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:04:14.472816 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:14.472449    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8785c58c-slgb8: secret "image-registry-tls" not found
Sep 16 12:04:14.472816 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:14.472448    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:14.472816 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:14.472497    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls podName:6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:30.472483309 +0000 UTC m=+67.226442018 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls") pod "image-registry-6f8785c58c-slgb8" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01") : secret "image-registry-tls" not found
Sep 16 12:04:14.472816 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:14.472509    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls podName:1f341464-bbd6-4a15-82fe-51149ded0036 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:30.472503121 +0000 UTC m=+67.226461830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls") pod "dns-default-xz5cp" (UID: "1f341464-bbd6-4a15-82fe-51149ded0036") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:21.563759 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.563725    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"]
Sep 16 12:04:21.594513 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.594477    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"]
Sep 16 12:04:21.594700 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.594605    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.596836 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.596810    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 12:04:21.596836 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.596843    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 12:04:21.597015 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.596879    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 12:04:21.597425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.597406    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 12:04:21.597425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.597415    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 12:04:21.597598 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.597421    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 12:04:21.597598 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.597409    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 12:04:21.623495 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.623469    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/60745e11-4f98-41fa-bf6b-8c9c58751509-hub\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.623665 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.623531    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/60745e11-4f98-41fa-bf6b-8c9c58751509-ca\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.623665 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.623565    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/60745e11-4f98-41fa-bf6b-8c9c58751509-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.623665 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.623606    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60745e11-4f98-41fa-bf6b-8c9c58751509-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.623785 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.623688    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/60745e11-4f98-41fa-bf6b-8c9c58751509-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.623785 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.623709    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtjs\" (UniqueName: \"kubernetes.io/projected/60745e11-4f98-41fa-bf6b-8c9c58751509-kube-api-access-7jtjs\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.725023 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.724988    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/60745e11-4f98-41fa-bf6b-8c9c58751509-hub\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.725184 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.725064    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/60745e11-4f98-41fa-bf6b-8c9c58751509-ca\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.725184 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.725085    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/60745e11-4f98-41fa-bf6b-8c9c58751509-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.725184 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.725105    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60745e11-4f98-41fa-bf6b-8c9c58751509-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.725184 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.725154    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/60745e11-4f98-41fa-bf6b-8c9c58751509-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.725184 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.725179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtjs\" (UniqueName: \"kubernetes.io/projected/60745e11-4f98-41fa-bf6b-8c9c58751509-kube-api-access-7jtjs\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.725876 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.725826    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/60745e11-4f98-41fa-bf6b-8c9c58751509-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.728521 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.728491    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/60745e11-4f98-41fa-bf6b-8c9c58751509-ca\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.728668 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.728560    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/60745e11-4f98-41fa-bf6b-8c9c58751509-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.728724 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.728667    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/60745e11-4f98-41fa-bf6b-8c9c58751509-hub\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.728764 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.728719    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60745e11-4f98-41fa-bf6b-8c9c58751509-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.732309 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.732287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtjs\" (UniqueName: \"kubernetes.io/projected/60745e11-4f98-41fa-bf6b-8c9c58751509-kube-api-access-7jtjs\") pod \"cluster-proxy-proxy-agent-5f4869fb46-bvtfj\" (UID: \"60745e11-4f98-41fa-bf6b-8c9c58751509\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:21.915269 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:21.915238    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:04:22.202646 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:22.202571    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"]
Sep 16 12:04:22.205695 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:04:22.205664    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60745e11_4f98_41fa_bf6b_8c9c58751509.slice/crio-3c2e229f1793a4753f244b4e67a4a68b3a4fbfdabfba5949ace1e897d29aed5f WatchSource:0}: Error finding container 3c2e229f1793a4753f244b4e67a4a68b3a4fbfdabfba5949ace1e897d29aed5f: Status 404 returned error can't find the container with id 3c2e229f1793a4753f244b4e67a4a68b3a4fbfdabfba5949ace1e897d29aed5f
Sep 16 12:04:23.024399 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:23.024353    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xflq6"
Sep 16 12:04:23.096521 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:23.096475    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj" event={"ID":"60745e11-4f98-41fa-bf6b-8c9c58751509","Type":"ContainerStarted","Data":"3c2e229f1793a4753f244b4e67a4a68b3a4fbfdabfba5949ace1e897d29aed5f"}
Sep 16 12:04:23.180092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:23.179925    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:23.242036 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:23.242000    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:23.657486 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:23.657448    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:23.709154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:23.709062    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:26.107056 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:26.107018    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj" event={"ID":"60745e11-4f98-41fa-bf6b-8c9c58751509","Type":"ContainerStarted","Data":"0479f3519cea923ca034fbb8bb15a50542d748a8423714d54cb714a4de564dd9"}
Sep 16 12:04:26.107427 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:26.107071    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj" event={"ID":"60745e11-4f98-41fa-bf6b-8c9c58751509","Type":"ContainerStarted","Data":"2aa8b2ff7482fdaa9bfbc3d5d4e0285b6c92bc117598198f1daf9d2e0829eee3"}
Sep 16 12:04:26.107427 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:26.107092    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj" event={"ID":"60745e11-4f98-41fa-bf6b-8c9c58751509","Type":"ContainerStarted","Data":"432864ec853717a3261fe15e7bca6bce47f5e75ef183857a5744f86ac6cfe25d"}
Sep 16 12:04:26.123362 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:26.123268    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj" podStartSLOduration=1.544505569 podStartE2EDuration="5.123248807s" totalImagesPullingTime="3.578743238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:26.122355793 +0000 UTC m=+62.876314524" watchObservedRunningTime="2026-09-16 12:04:26.123248807 +0000 UTC m=+62.877207539"
Sep 16 12:04:27.171278 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:27.171243    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:27.227840 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:27.227808    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:27.281602 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:27.281566    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:29.388215 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:29.388180    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:04:29.390541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:29.390519    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:04:29.399173 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:29.399157    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:04:29.399269 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:29.399222    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs podName:1617d1a8-1f68-4475-88f9-f6fdb91a07db nodeName:}" failed. No retries permitted until 2026-09-16 12:05:33.399201171 +0000 UTC m=+130.153159880 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs") pod "network-metrics-daemon-pm6d7" (UID: "1617d1a8-1f68-4475-88f9-f6fdb91a07db") : secret "metrics-daemon-secret" not found
Sep 16 12:04:29.489313 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:29.489262    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:04:29.491635 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:29.491612    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:04:29.502594 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:29.502565    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82e3f885-f60e-4039-ac7d-3318d0fb27ca-original-pull-secret\") pod \"global-pull-secret-syncer-xc9vl\" (UID: \"82e3f885-f60e-4039-ac7d-3318d0fb27ca\") " pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:04:29.590300 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:29.590254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29zz\" (UniqueName: \"kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz\") pod \"network-check-target-bzbgg\" (UID: \"cca7ed59-7144-44f8-8421-497cb96f2aae\") " pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:04:29.592781 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:29.592763    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:04:29.602940 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:29.602922    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:04:29.613661 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:29.613638    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d29zz\" (UniqueName: \"kubernetes.io/projected/cca7ed59-7144-44f8-8421-497cb96f2aae-kube-api-access-d29zz\") pod \"network-check-target-bzbgg\" (UID: \"cca7ed59-7144-44f8-8421-497cb96f2aae\") " pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:04:29.766225 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:29.766153    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gvq92\""
Sep 16 12:04:29.774944 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:29.774922    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:04:29.778199 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:29.778180    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9vl"
Sep 16 12:04:30.045585 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:30.045499    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xc9vl"]
Sep 16 12:04:30.048369 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:04:30.048344    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e3f885_f60e_4039_ac7d_3318d0fb27ca.slice/crio-d99468ab2a5484b92487c12f363effc148f85486273ec0c6cc435c2bfaaa411f WatchSource:0}: Error finding container d99468ab2a5484b92487c12f363effc148f85486273ec0c6cc435c2bfaaa411f: Status 404 returned error can't find the container with id d99468ab2a5484b92487c12f363effc148f85486273ec0c6cc435c2bfaaa411f
Sep 16 12:04:30.065900 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:30.065873    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bzbgg"]
Sep 16 12:04:30.068297 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:04:30.068271    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca7ed59_7144_44f8_8421_497cb96f2aae.slice/crio-84eff4aa7e7e74033a248c7c43d3a80c998c1e247e909a537babcb54f7339628 WatchSource:0}: Error finding container 84eff4aa7e7e74033a248c7c43d3a80c998c1e247e909a537babcb54f7339628: Status 404 returned error can't find the container with id 84eff4aa7e7e74033a248c7c43d3a80c998c1e247e909a537babcb54f7339628
Sep 16 12:04:30.120325 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:30.120284    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xc9vl" event={"ID":"82e3f885-f60e-4039-ac7d-3318d0fb27ca","Type":"ContainerStarted","Data":"d99468ab2a5484b92487c12f363effc148f85486273ec0c6cc435c2bfaaa411f"}
Sep 16 12:04:30.121320 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:30.121296    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bzbgg" event={"ID":"cca7ed59-7144-44f8-8421-497cb96f2aae","Type":"ContainerStarted","Data":"84eff4aa7e7e74033a248c7c43d3a80c998c1e247e909a537babcb54f7339628"}
Sep 16 12:04:30.497136 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:30.497101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:04:30.497566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:30.497160    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:04:30.497566 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:30.497262    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:30.497566 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:30.497265    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:04:30.497566 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:30.497361    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8785c58c-slgb8: secret "image-registry-tls" not found
Sep 16 12:04:30.497566 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:30.497369    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls podName:1f341464-bbd6-4a15-82fe-51149ded0036 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:02.497347698 +0000 UTC m=+99.251306424 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls") pod "dns-default-xz5cp" (UID: "1f341464-bbd6-4a15-82fe-51149ded0036") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:30.497566 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:04:30.497501    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls podName:6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:02.497488764 +0000 UTC m=+99.251447480 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls") pod "image-registry-6f8785c58c-slgb8" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01") : secret "image-registry-tls" not found
Sep 16 12:04:35.137057 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:35.137017    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bzbgg" event={"ID":"cca7ed59-7144-44f8-8421-497cb96f2aae","Type":"ContainerStarted","Data":"ae957d585672dc17a0416ca4ea0df07b1e543fe2f20c862970be8c597cd00762"}
Sep 16 12:04:35.137491 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:35.137074    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:04:35.138521 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:35.138499    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xc9vl" event={"ID":"82e3f885-f60e-4039-ac7d-3318d0fb27ca","Type":"ContainerStarted","Data":"57b262b19c16b1987116798bfa4c4fbdd99b3e08689133793f6d82292a8209a4"}
Sep 16 12:04:35.153692 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:35.153647    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bzbgg" podStartSLOduration=66.706240214 podStartE2EDuration="1m11.153634852s" totalImagesPullingTime="4.447394638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:35.152079978 +0000 UTC m=+71.906038709" watchObservedRunningTime="2026-09-16 12:04:35.153634852 +0000 UTC m=+71.907593622"
Sep 16 12:04:35.166977 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:35.166935    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xc9vl" podStartSLOduration=66.695035408 podStartE2EDuration="1m11.166922657s" totalImagesPullingTime="4.471887249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:35.166893194 +0000 UTC m=+71.920851925" watchObservedRunningTime="2026-09-16 12:04:35.166922657 +0000 UTC m=+71.920881394"
Sep 16 12:04:37.898249 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:37.898219    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:38.002376 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:38.002342    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:59.214691 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:04:59.214575    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:05:02.523511 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:05:02.523470    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:05:02.524030 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:05:02.523539    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:05:02.524030 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:05:02.523624    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:05:02.524030 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:05:02.523640    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:05:02.524030 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:05:02.523645    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8785c58c-slgb8: secret "image-registry-tls" not found
Sep 16 12:05:02.524030 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:05:02.523703    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls podName:1f341464-bbd6-4a15-82fe-51149ded0036 nodeName:}" failed. No retries permitted until 2026-09-16 12:06:06.523689937 +0000 UTC m=+163.277648646 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls") pod "dns-default-xz5cp" (UID: "1f341464-bbd6-4a15-82fe-51149ded0036") : secret "dns-default-metrics-tls" not found
Sep 16 12:05:02.524030 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:05:02.523717    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls podName:6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01 nodeName:}" failed. No retries permitted until 2026-09-16 12:06:06.52371159 +0000 UTC m=+163.277670299 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls") pod "image-registry-6f8785c58c-slgb8" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01") : secret "image-registry-tls" not found
Sep 16 12:05:02.770345 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:05:02.770313    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:05:06.143769 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:05:06.143735    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bzbgg"
Sep 16 12:05:33.433515 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:05:33.433478    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:05:33.434022 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:05:33.433643    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:05:33.434022 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:05:33.433699    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs podName:1617d1a8-1f68-4475-88f9-f6fdb91a07db nodeName:}" failed. No retries permitted until 2026-09-16 12:07:35.433684826 +0000 UTC m=+252.187643540 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs") pod "network-metrics-daemon-pm6d7" (UID: "1617d1a8-1f68-4475-88f9-f6fdb91a07db") : secret "metrics-daemon-secret" not found
Sep 16 12:06:01.594286 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:06:01.594241    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8" podUID="6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01"
Sep 16 12:06:01.610453 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:06:01.610414    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xz5cp" podUID="1f341464-bbd6-4a15-82fe-51149ded0036"
Sep 16 12:06:02.380430 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:06:02.380400    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:06:02.380612 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:06:02.380400    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:06:02.771673 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:06:02.771584    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:06:06.568464 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:06:06.568430    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:06:06.568464 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:06:06.568468    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:06:06.569035 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:06:06.568591    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:06:06.569035 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:06:06.568602    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8785c58c-slgb8: secret "image-registry-tls" not found
Sep 16 12:06:06.569035 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:06:06.568611    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:06:06.569035 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:06:06.568647    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls podName:6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:08.56863461 +0000 UTC m=+285.322593320 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls") pod "image-registry-6f8785c58c-slgb8" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01") : secret "image-registry-tls" not found
Sep 16 12:06:06.569035 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:06:06.568677    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls podName:1f341464-bbd6-4a15-82fe-51149ded0036 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:08.568659416 +0000 UTC m=+285.322618144 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls") pod "dns-default-xz5cp" (UID: "1f341464-bbd6-4a15-82fe-51149ded0036") : secret "dns-default-metrics-tls" not found
Sep 16 12:06:15.755085 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:06:15.755054    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:07:35.488531 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:35.488494    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:07:35.489021 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:35.488643    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:07:35.489021 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:35.488709    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs podName:1617d1a8-1f68-4475-88f9-f6fdb91a07db nodeName:}" failed. No retries permitted until 2026-09-16 12:09:37.48869396 +0000 UTC m=+374.242652673 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs") pod "network-metrics-daemon-pm6d7" (UID: "1617d1a8-1f68-4475-88f9-f6fdb91a07db") : secret "metrics-daemon-secret" not found
Sep 16 12:07:53.055172 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:53.055094    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:53.140442 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:53.140412    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:54.656855 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:54.656818    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:54.758793 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:54.758761    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:56.360400 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.360367    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:56.870227 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.870195    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ts76n"]
Sep 16 12:07:56.873040 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.873018    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:56.874896 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.874875    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:56.875247 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.875227    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 12:07:56.875352 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.875286    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 12:07:56.875352 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.875302    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 12:07:56.876068 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.876054    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 12:07:56.876123 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.876103    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hqfk\""
Sep 16 12:07:56.885445 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.885426    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ts76n"]
Sep 16 12:07:56.931242 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.931213    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e51c0fd-41e1-40df-9051-d41529de1881-data-volume\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:56.931242 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.931242    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88pdq\" (UniqueName: \"kubernetes.io/projected/1e51c0fd-41e1-40df-9051-d41529de1881-kube-api-access-88pdq\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:56.931431 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.931265    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e51c0fd-41e1-40df-9051-d41529de1881-crio-socket\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:56.931431 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.931281    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:56.931431 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.931368    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:56.950238 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:56.950214    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:57.032285 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:57.032263    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e51c0fd-41e1-40df-9051-d41529de1881-data-volume\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:57.032379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:57.032289    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88pdq\" (UniqueName: \"kubernetes.io/projected/1e51c0fd-41e1-40df-9051-d41529de1881-kube-api-access-88pdq\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:57.032379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:57.032311    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e51c0fd-41e1-40df-9051-d41529de1881-crio-socket\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:57.032379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:57.032332    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:57.032488 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:57.032410    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e51c0fd-41e1-40df-9051-d41529de1881-crio-socket\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:57.032488 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:57.032417    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:57.032488 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:57.032475    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:57.032621 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:57.032501    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls podName:1e51c0fd-41e1-40df-9051-d41529de1881 nodeName:}" failed. No retries permitted until 2026-09-16 12:07:57.532481072 +0000 UTC m=+274.286439787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ts76n" (UID: "1e51c0fd-41e1-40df-9051-d41529de1881") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:57.032621 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:57.032530    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:07:57.032621 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:57.032597    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e51c0fd-41e1-40df-9051-d41529de1881-data-volume\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:57.032731 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:57.032605    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm podName:1e51c0fd-41e1-40df-9051-d41529de1881 nodeName:}" failed. No retries permitted until 2026-09-16 12:07:57.532594183 +0000 UTC m=+274.286552898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ts76n" (UID: "1e51c0fd-41e1-40df-9051-d41529de1881") : configmap "kube-rbac-proxy" not found
Sep 16 12:07:57.049505 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:57.049488    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88pdq\" (UniqueName: \"kubernetes.io/projected/1e51c0fd-41e1-40df-9051-d41529de1881-kube-api-access-88pdq\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:57.095366 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:57.095341    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:57.160970 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:57.160881    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:57.214565 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:57.214523    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:57.536019 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:57.535940    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:57.536019 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:57.535980    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:57.536369 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:57.536054    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:07:57.536369 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:57.536083    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:57.536369 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:57.536102    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm podName:1e51c0fd-41e1-40df-9051-d41529de1881 nodeName:}" failed. No retries permitted until 2026-09-16 12:07:58.536090557 +0000 UTC m=+275.290049266 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ts76n" (UID: "1e51c0fd-41e1-40df-9051-d41529de1881") : configmap "kube-rbac-proxy" not found
Sep 16 12:07:57.536369 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:57.536141    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls podName:1e51c0fd-41e1-40df-9051-d41529de1881 nodeName:}" failed. No retries permitted until 2026-09-16 12:07:58.536129444 +0000 UTC m=+275.290088179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ts76n" (UID: "1e51c0fd-41e1-40df-9051-d41529de1881") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:58.134629 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:58.134595    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:58.203170 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:58.203145    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:58.545238 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:58.545134    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:58.545238 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:58.545206    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:07:58.545705 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:58.545278    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:58.545705 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:58.545308    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:07:58.545705 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:58.545342    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls podName:1e51c0fd-41e1-40df-9051-d41529de1881 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:00.545325896 +0000 UTC m=+277.299284609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ts76n" (UID: "1e51c0fd-41e1-40df-9051-d41529de1881") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:58.545705 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:07:58.545376    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm podName:1e51c0fd-41e1-40df-9051-d41529de1881 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:00.545358019 +0000 UTC m=+277.299316758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ts76n" (UID: "1e51c0fd-41e1-40df-9051-d41529de1881") : configmap "kube-rbac-proxy" not found
Sep 16 12:07:58.562951 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:58.562919    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:58.638191 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:58.638174    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:58.827792 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:58.827728    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fk9r5_678bc08c-e65a-4d2f-8de1-5b73269448c2/dns-node-resolver/0.log"
Sep 16 12:07:58.839950 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:58.839928    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:58.940284 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:58.940259    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:59.816921 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:07:59.816888    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:00.028269 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:00.028239    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tgspg_4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b/node-ca/0.log"
Sep 16 12:08:00.379265 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:00.379239    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:00.470220 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:00.470191    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:00.526102 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:00.526073    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:00.560394 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:00.560373    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:08:00.560471 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:00.560414    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:08:00.560507 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:00.560483    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:08:00.560541 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:00.560512    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:00.560541 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:00.560529    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm podName:1e51c0fd-41e1-40df-9051-d41529de1881 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:04.560517725 +0000 UTC m=+281.314476433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ts76n" (UID: "1e51c0fd-41e1-40df-9051-d41529de1881") : configmap "kube-rbac-proxy" not found
Sep 16 12:08:00.560638 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:00.560577    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls podName:1e51c0fd-41e1-40df-9051-d41529de1881 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:04.560558655 +0000 UTC m=+281.314517440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ts76n" (UID: "1e51c0fd-41e1-40df-9051-d41529de1881") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:00.827833 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:00.827758    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sshpk_3ef9d7a4-3c9d-4802-8871-18079adb56ea/serve-healthcheck-canary/0.log"
Sep 16 12:08:02.058512 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:02.058472    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:03.863841 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:03.863809    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:04.017000 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:04.016965    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:04.109755 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:04.109731    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:04.199009 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:04.198929    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:04.312326 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:04.312301    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:04.454526 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:04.454435    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:04.588557 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:04.588508    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:08:04.588721 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:04.588588    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:08:04.588721 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:04.588658    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:04.588721 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:04.588706    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:08:04.588828 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:04.588725    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls podName:1e51c0fd-41e1-40df-9051-d41529de1881 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:12.588710339 +0000 UTC m=+289.342669053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ts76n" (UID: "1e51c0fd-41e1-40df-9051-d41529de1881") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:04.588828 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:04.588765    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm podName:1e51c0fd-41e1-40df-9051-d41529de1881 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:12.588753133 +0000 UTC m=+289.342711849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ts76n" (UID: "1e51c0fd-41e1-40df-9051-d41529de1881") : configmap "kube-rbac-proxy" not found
Sep 16 12:08:04.607108 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:04.607090    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:04.786038 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:04.785926    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:05.381601 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:05.381531    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xz5cp" podUID="1f341464-bbd6-4a15-82fe-51149ded0036"
Sep 16 12:08:05.381601 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:05.381568    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8" podUID="6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01"
Sep 16 12:08:05.600172 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:05.600124    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:05.708318 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:05.708235    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:08:05.708458 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:05.708237    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:08:05.983663 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:05.983567    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:06.177997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:06.177950    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:06.371262 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:06.371235    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:06.766011 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:06.765896    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:07.362732 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:07.362699    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:07.760253 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:07.760136    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:07.828798 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:07.828770    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:08.175609 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:08.175542    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:08.370787 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:08.370753    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:08.618289 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:08.618255    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:08:08.618450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:08.618304    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:08:08.620589 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:08.620564    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f341464-bbd6-4a15-82fe-51149ded0036-metrics-tls\") pod \"dns-default-xz5cp\" (UID: \"1f341464-bbd6-4a15-82fe-51149ded0036\") " pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:08:08.620701 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:08.620652    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") pod \"image-registry-6f8785c58c-slgb8\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") " pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:08:08.711944 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:08.711921    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-z95hj\""
Sep 16 12:08:08.711944 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:08.711930    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7d4qr\""
Sep 16 12:08:08.719246 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:08.719223    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:08:08.719246 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:08.719244    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:08:08.758563 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:08.758514    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:09.000935 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:09.000859    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xz5cp"]
Sep 16 12:08:09.003213 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:09.003190    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f8785c58c-slgb8"]
Sep 16 12:08:09.004207 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:08:09.004183    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f341464_bbd6_4a15_82fe_51149ded0036.slice/crio-1a439b35ba01fb09b4a9412a3991f147bf6f92b20a247a775d8e7a69baf27530 WatchSource:0}: Error finding container 1a439b35ba01fb09b4a9412a3991f147bf6f92b20a247a775d8e7a69baf27530: Status 404 returned error can't find the container with id 1a439b35ba01fb09b4a9412a3991f147bf6f92b20a247a775d8e7a69baf27530
Sep 16 12:08:09.006818 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:08:09.006796    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9a48ef_d9c6_4ea9_ad86_6f1a9efaba01.slice/crio-e38095546a13b07c72292be72f0d819513b805228f91677e38887e76e18eafd7 WatchSource:0}: Error finding container e38095546a13b07c72292be72f0d819513b805228f91677e38887e76e18eafd7: Status 404 returned error can't find the container with id e38095546a13b07c72292be72f0d819513b805228f91677e38887e76e18eafd7
Sep 16 12:08:09.367810 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:09.367779    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:09.720964 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:09.720831    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8" event={"ID":"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01","Type":"ContainerStarted","Data":"17f4d89f1e7d2de8ec7317cb1b2778fcf72a8d38ee9f20efdb21cbe4c3443a14"}
Sep 16 12:08:09.721130 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:09.720992    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:08:09.721130 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:09.721030    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8" event={"ID":"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01","Type":"ContainerStarted","Data":"e38095546a13b07c72292be72f0d819513b805228f91677e38887e76e18eafd7"}
Sep 16 12:08:09.722264 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:09.722240    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xz5cp" event={"ID":"1f341464-bbd6-4a15-82fe-51149ded0036","Type":"ContainerStarted","Data":"1a439b35ba01fb09b4a9412a3991f147bf6f92b20a247a775d8e7a69baf27530"}
Sep 16 12:08:09.744619 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:09.744575    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8" podStartSLOduration=285.74456113 podStartE2EDuration="4m45.74456113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:09.742101318 +0000 UTC m=+286.496060049" watchObservedRunningTime="2026-09-16 12:08:09.74456113 +0000 UTC m=+286.498519851"
Sep 16 12:08:09.780819 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:09.780792    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:09.959292 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:09.959255    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:10.088768 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:10.088674    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:10.419817 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:10.419777    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:10.661194 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:10.661148    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:10.960962 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:10.960928    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:11.362979 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:11.362951    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:11.729631 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:11.729539    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xz5cp" event={"ID":"1f341464-bbd6-4a15-82fe-51149ded0036","Type":"ContainerStarted","Data":"4c05b36319fe456ecc4f7dc66e5add8efdc1992ced6ad933345a82d8180c23b3"}
Sep 16 12:08:11.729631 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:11.729601    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xz5cp" event={"ID":"1f341464-bbd6-4a15-82fe-51149ded0036","Type":"ContainerStarted","Data":"fa41eb749c2f6a252f1a3b6813e5a95db836be8f107b51b5c37d50c9392d86ac"}
Sep 16 12:08:11.729803 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:11.729693    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:08:11.746172 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:11.746129    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xz5cp" podStartSLOduration=252.014211459 podStartE2EDuration="4m13.746115608s" totalImagesPullingTime="1.731904149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:11.745687645 +0000 UTC m=+288.499646377" watchObservedRunningTime="2026-09-16 12:08:11.746115608 +0000 UTC m=+288.500074338"
Sep 16 12:08:11.762256 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:11.762235    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:11.964291 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:11.964255    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:12.559636 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:12.559606    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:12.647314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:12.647284    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:08:12.647474 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:12.647324    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:08:12.647474 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:12.647446    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:12.647606 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:12.647527    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls podName:1e51c0fd-41e1-40df-9051-d41529de1881 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:28.64750558 +0000 UTC m=+305.401464295 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ts76n" (UID: "1e51c0fd-41e1-40df-9051-d41529de1881") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:12.647792 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:12.647775    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e51c0fd-41e1-40df-9051-d41529de1881-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:08:12.965462 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:12.965426    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:13.134644 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:13.134613    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:13.530486 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:13.530454    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:13.627948 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:13.627919    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:14.164491 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:14.164461    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:14.360719 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:14.360691    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:14.563866 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:14.563783    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:18.508656 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:18.508626    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:18.612320 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:18.612293    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:18.756007 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:18.755970    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-pm6d7" podUID="1617d1a8-1f68-4475-88f9-f6fdb91a07db"
Sep 16 12:08:18.756766 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:18.756743    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:18.863302 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:18.863271    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:19.210635 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:19.210570    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:19.444334 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:19.444303    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:19.523894 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:19.523814    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:19.950447 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:19.950416    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:20.008866 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:20.008837    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:20.460388 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:20.460356    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:20.635228 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:20.635184    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:20.692588 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:20.692539    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:21.271741 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:21.271696    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:21.333020 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:21.332991    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:21.734658 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:21.734631    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xz5cp"
Sep 16 12:08:23.168003 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.167975    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f8785c58c-slgb8"]
Sep 16 12:08:23.255014 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.254983    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:23.258783 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.258759    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68fc5bf8bf-4d95z"]
Sep 16 12:08:23.262865 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.262849    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.274642 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.274622    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68fc5bf8bf-4d95z"]
Sep 16 12:08:23.322384 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.322354    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22f72982-d747-49aa-81e6-f274ce50bb39-trusted-ca\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.322493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.322403    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22f72982-d747-49aa-81e6-f274ce50bb39-installation-pull-secrets\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.322493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.322449    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f72982-d747-49aa-81e6-f274ce50bb39-registry-tls\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.322493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.322481    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22f72982-d747-49aa-81e6-f274ce50bb39-bound-sa-token\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.322669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.322497    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhhcl\" (UniqueName: \"kubernetes.io/projected/22f72982-d747-49aa-81e6-f274ce50bb39-kube-api-access-hhhcl\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.322669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.322512    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22f72982-d747-49aa-81e6-f274ce50bb39-registry-certificates\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.322669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.322602    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22f72982-d747-49aa-81e6-f274ce50bb39-image-registry-private-configuration\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.322669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.322636    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22f72982-d747-49aa-81e6-f274ce50bb39-ca-trust-extracted\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.347991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.347967    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:23.423340 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.423288    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22f72982-d747-49aa-81e6-f274ce50bb39-image-registry-private-configuration\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.423340 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.423316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22f72982-d747-49aa-81e6-f274ce50bb39-ca-trust-extracted\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.423443 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.423354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22f72982-d747-49aa-81e6-f274ce50bb39-trusted-ca\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.423505 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.423486    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22f72982-d747-49aa-81e6-f274ce50bb39-installation-pull-secrets\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.423579 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.423521    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f72982-d747-49aa-81e6-f274ce50bb39-registry-tls\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.423689 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.423665    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22f72982-d747-49aa-81e6-f274ce50bb39-bound-sa-token\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.423805 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.423702    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhhcl\" (UniqueName: \"kubernetes.io/projected/22f72982-d747-49aa-81e6-f274ce50bb39-kube-api-access-hhhcl\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.423805 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.423720    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22f72982-d747-49aa-81e6-f274ce50bb39-ca-trust-extracted\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.423805 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.423733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22f72982-d747-49aa-81e6-f274ce50bb39-registry-certificates\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.424279 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.424258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22f72982-d747-49aa-81e6-f274ce50bb39-trusted-ca\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.424451 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.424428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22f72982-d747-49aa-81e6-f274ce50bb39-registry-certificates\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.425822 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.425799    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22f72982-d747-49aa-81e6-f274ce50bb39-image-registry-private-configuration\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.426078 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.426059    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22f72982-d747-49aa-81e6-f274ce50bb39-installation-pull-secrets\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.426122 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.426060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f72982-d747-49aa-81e6-f274ce50bb39-registry-tls\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.430734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.430707    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22f72982-d747-49aa-81e6-f274ce50bb39-bound-sa-token\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.430827 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.430709    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhhcl\" (UniqueName: \"kubernetes.io/projected/22f72982-d747-49aa-81e6-f274ce50bb39-kube-api-access-hhhcl\") pod \"image-registry-68fc5bf8bf-4d95z\" (UID: \"22f72982-d747-49aa-81e6-f274ce50bb39\") " pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.570895 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.570866    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:23.598087 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.598065    2558 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 12:08:23.847697 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:23.847674    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68fc5bf8bf-4d95z"]
Sep 16 12:08:23.857171 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:08:23.850795    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f72982_d747_49aa_81e6_f274ce50bb39.slice/crio-dcbd990e91159335d82db4cc783043a5eea758569acb12a742eb776e13eaa095 WatchSource:0}: Error finding container dcbd990e91159335d82db4cc783043a5eea758569acb12a742eb776e13eaa095: Status 404 returned error can't find the container with id dcbd990e91159335d82db4cc783043a5eea758569acb12a742eb776e13eaa095
Sep 16 12:08:24.765930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:24.765894    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z" event={"ID":"22f72982-d747-49aa-81e6-f274ce50bb39","Type":"ContainerStarted","Data":"bdef56c84aa34f696bb243018473939836120066b8136f416f30370a22fd010b"}
Sep 16 12:08:24.765930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:24.765938    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z" event={"ID":"22f72982-d747-49aa-81e6-f274ce50bb39","Type":"ContainerStarted","Data":"dcbd990e91159335d82db4cc783043a5eea758569acb12a742eb776e13eaa095"}
Sep 16 12:08:24.766327 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:24.765957    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:24.786720 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:24.786677    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z" podStartSLOduration=1.786661915 podStartE2EDuration="1.786661915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:24.785232979 +0000 UTC m=+301.539191709" watchObservedRunningTime="2026-09-16 12:08:24.786661915 +0000 UTC m=+301.540620646"
Sep 16 12:08:27.259417 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:27.259384    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:28.664453 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:28.664415    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:08:28.666719 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:28.666700    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e51c0fd-41e1-40df-9051-d41529de1881-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts76n\" (UID: \"1e51c0fd-41e1-40df-9051-d41529de1881\") " pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:08:28.683837 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:28.683815    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hqfk\""
Sep 16 12:08:28.692432 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:28.692417    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ts76n"
Sep 16 12:08:28.965107 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:28.964918    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ts76n"]
Sep 16 12:08:28.967563 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:08:28.967520    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e51c0fd_41e1_40df_9051_d41529de1881.slice/crio-d173f3090c0c9b01f0349f9a1772de55f2f85f64c89d2f6e73f1292443a811aa WatchSource:0}: Error finding container d173f3090c0c9b01f0349f9a1772de55f2f85f64c89d2f6e73f1292443a811aa: Status 404 returned error can't find the container with id d173f3090c0c9b01f0349f9a1772de55f2f85f64c89d2f6e73f1292443a811aa
Sep 16 12:08:29.024236 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:29.024217    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:08:29.046598 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:29.046575    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:29.413686 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:29.413653    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:29.514124 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:29.514084    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:29.779832 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:29.779804    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ts76n" event={"ID":"1e51c0fd-41e1-40df-9051-d41529de1881","Type":"ContainerStarted","Data":"c4d35a67ba3c4229881a562e71e5da329f56775706244b6bc1d405be64b4b1b9"}
Sep 16 12:08:29.780144 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:29.779856    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ts76n" event={"ID":"1e51c0fd-41e1-40df-9051-d41529de1881","Type":"ContainerStarted","Data":"6cb9d94770e803493f2bcfcc2d773a3a6339cb89e8100f1bbbe88dd06469223a"}
Sep 16 12:08:29.780144 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:29.779876    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ts76n" event={"ID":"1e51c0fd-41e1-40df-9051-d41529de1881","Type":"ContainerStarted","Data":"d173f3090c0c9b01f0349f9a1772de55f2f85f64c89d2f6e73f1292443a811aa"}
Sep 16 12:08:31.283075 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:31.283048    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:31.381691 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:31.381611    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:31.788291 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:31.788209    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ts76n" event={"ID":"1e51c0fd-41e1-40df-9051-d41529de1881","Type":"ContainerStarted","Data":"ebf7894abc7abd5397bbed185de3ef752ceeb33f2d3780dd513693e95c5a2783"}
Sep 16 12:08:31.806311 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:31.806268    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ts76n" podStartSLOduration=33.790929061 podStartE2EDuration="35.806253976s" totalImagesPullingTime="2.015324915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:07:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:31.804777721 +0000 UTC m=+308.558736449" watchObservedRunningTime="2026-09-16 12:08:31.806253976 +0000 UTC m=+308.560212706"
Sep 16 12:08:32.201182 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:32.201151    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:32.300473 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:32.300448    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:32.435058 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:32.435029    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:32.501260 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:32.501194    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:32.890162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:32.890137    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:33.173295 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:33.173227    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:08:33.752407 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:33.752376    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:08:34.193264 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:34.193234    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:34.252934 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:34.252905    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:35.869490 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:35.869460    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:37.643279 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.643253    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:37.738032 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.738005    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:37.820110 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.820081    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"]
Sep 16 12:08:37.823279 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.823264    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:37.825598 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.825579    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 12:08:37.825729 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.825711    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 12:08:37.826361 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.826347    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 12:08:37.826470 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.826453    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 12:08:37.826521 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.826473    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 12:08:37.826584 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.826540    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vqm29\""
Sep 16 12:08:37.831449 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.831431    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:37.837194 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.837170    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"]
Sep 16 12:08:37.874273 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.874249    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vcpsg"]
Sep 16 12:08:37.877244 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.877228    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:37.879297 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.879280    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 12:08:37.879395 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.879297    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rvlt5\""
Sep 16 12:08:37.879395 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.879379    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 12:08:37.879889 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.879875    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 12:08:37.933147 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.933086    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11685ce0-7e31-48ff-b729-dfe81f8cfcde-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4cvlw\" (UID: \"11685ce0-7e31-48ff-b729-dfe81f8cfcde\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:37.933147 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.933124    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11685ce0-7e31-48ff-b729-dfe81f8cfcde-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4cvlw\" (UID: \"11685ce0-7e31-48ff-b729-dfe81f8cfcde\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:37.933314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.933207    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m98mv\" (UniqueName: \"kubernetes.io/projected/11685ce0-7e31-48ff-b729-dfe81f8cfcde-kube-api-access-m98mv\") pod \"openshift-state-metrics-67dcf8c979-4cvlw\" (UID: \"11685ce0-7e31-48ff-b729-dfe81f8cfcde\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:37.933314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.933271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/11685ce0-7e31-48ff-b729-dfe81f8cfcde-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4cvlw\" (UID: \"11685ce0-7e31-48ff-b729-dfe81f8cfcde\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:37.965249 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:37.965224    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:38.034444 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.034420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.034578 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.034448    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f106348d-5939-4254-a633-a5f44e38b5bf-root\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.034578 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.034472    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/11685ce0-7e31-48ff-b729-dfe81f8cfcde-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4cvlw\" (UID: \"11685ce0-7e31-48ff-b729-dfe81f8cfcde\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:38.034667 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.034581    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.034667 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.034617    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11685ce0-7e31-48ff-b729-dfe81f8cfcde-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4cvlw\" (UID: \"11685ce0-7e31-48ff-b729-dfe81f8cfcde\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:38.034667 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.034639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11685ce0-7e31-48ff-b729-dfe81f8cfcde-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4cvlw\" (UID: \"11685ce0-7e31-48ff-b729-dfe81f8cfcde\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:38.034772 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.034686    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f106348d-5939-4254-a633-a5f44e38b5bf-metrics-client-ca\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.034772 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.034706    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-tls\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.034772 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.034728    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-wtmp\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.034772 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.034748    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f106348d-5939-4254-a633-a5f44e38b5bf-sys\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.034939 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.034779    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbf5t\" (UniqueName: \"kubernetes.io/projected/f106348d-5939-4254-a633-a5f44e38b5bf-kube-api-access-fbf5t\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.034939 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.034840    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-textfile\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.034939 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.034867    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98mv\" (UniqueName: \"kubernetes.io/projected/11685ce0-7e31-48ff-b729-dfe81f8cfcde-kube-api-access-m98mv\") pod \"openshift-state-metrics-67dcf8c979-4cvlw\" (UID: \"11685ce0-7e31-48ff-b729-dfe81f8cfcde\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:38.035347 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.035326    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11685ce0-7e31-48ff-b729-dfe81f8cfcde-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4cvlw\" (UID: \"11685ce0-7e31-48ff-b729-dfe81f8cfcde\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:38.036961 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.036941    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11685ce0-7e31-48ff-b729-dfe81f8cfcde-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4cvlw\" (UID: \"11685ce0-7e31-48ff-b729-dfe81f8cfcde\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:38.037059 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.037012    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/11685ce0-7e31-48ff-b729-dfe81f8cfcde-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4cvlw\" (UID: \"11685ce0-7e31-48ff-b729-dfe81f8cfcde\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:38.048832 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.048812    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m98mv\" (UniqueName: \"kubernetes.io/projected/11685ce0-7e31-48ff-b729-dfe81f8cfcde-kube-api-access-m98mv\") pod \"openshift-state-metrics-67dcf8c979-4cvlw\" (UID: \"11685ce0-7e31-48ff-b729-dfe81f8cfcde\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:38.065875 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.065853    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:38.132854 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.132831    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"
Sep 16 12:08:38.135732 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.135710    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.135740    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f106348d-5939-4254-a633-a5f44e38b5bf-root\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.135769    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.135811    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f106348d-5939-4254-a633-a5f44e38b5bf-metrics-client-ca\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.135819    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f106348d-5939-4254-a633-a5f44e38b5bf-root\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.135838    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-tls\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.135857    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-wtmp\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.135935    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f106348d-5939-4254-a633-a5f44e38b5bf-sys\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.135964    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbf5t\" (UniqueName: \"kubernetes.io/projected/f106348d-5939-4254-a633-a5f44e38b5bf-kube-api-access-fbf5t\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.135984    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-wtmp\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.136015    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-textfile\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.136030    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f106348d-5939-4254-a633-a5f44e38b5bf-sys\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136786 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.136409    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-textfile\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136786 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.136464    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.136786 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.136517    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f106348d-5939-4254-a633-a5f44e38b5bf-metrics-client-ca\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.138015 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.137996    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.138211 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.138188    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f106348d-5939-4254-a633-a5f44e38b5bf-node-exporter-tls\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.146078 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.146054    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbf5t\" (UniqueName: \"kubernetes.io/projected/f106348d-5939-4254-a633-a5f44e38b5bf-kube-api-access-fbf5t\") pod \"node-exporter-vcpsg\" (UID: \"f106348d-5939-4254-a633-a5f44e38b5bf\") " pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.185942 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.185851    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vcpsg"
Sep 16 12:08:38.199042 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:08:38.199003    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf106348d_5939_4254_a633_a5f44e38b5bf.slice/crio-2de76707f84d5386e2f935be5659a25368ada001a0aed1edaa69f47350050d7a WatchSource:0}: Error finding container 2de76707f84d5386e2f935be5659a25368ada001a0aed1edaa69f47350050d7a: Status 404 returned error can't find the container with id 2de76707f84d5386e2f935be5659a25368ada001a0aed1edaa69f47350050d7a
Sep 16 12:08:38.246025 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.245989    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:38.343509 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.343476    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:38.410289 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.410266    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw"]
Sep 16 12:08:38.412315 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:08:38.412291    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11685ce0_7e31_48ff_b729_dfe81f8cfcde.slice/crio-ce259a95309d6209adfb1feea17a38e051935a652015e21ad1701d4e093a22fa WatchSource:0}: Error finding container ce259a95309d6209adfb1feea17a38e051935a652015e21ad1701d4e093a22fa: Status 404 returned error can't find the container with id ce259a95309d6209adfb1feea17a38e051935a652015e21ad1701d4e093a22fa
Sep 16 12:08:38.521002 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.520970    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:38.620077 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.620038    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:38.811452 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.811370    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw" event={"ID":"11685ce0-7e31-48ff-b729-dfe81f8cfcde","Type":"ContainerStarted","Data":"6e492627280cecb4a3bb787908f5ce7ae800c34b060751b6f0098aaa89d0620b"}
Sep 16 12:08:38.811917 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.811470    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw" event={"ID":"11685ce0-7e31-48ff-b729-dfe81f8cfcde","Type":"ContainerStarted","Data":"0f25ee07b70c56aac54df617d2517000dd4a71d68dc28d87d31bd6b4bb692a0a"}
Sep 16 12:08:38.811917 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.811499    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw" event={"ID":"11685ce0-7e31-48ff-b729-dfe81f8cfcde","Type":"ContainerStarted","Data":"ce259a95309d6209adfb1feea17a38e051935a652015e21ad1701d4e093a22fa"}
Sep 16 12:08:38.812660 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.812636    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vcpsg" event={"ID":"f106348d-5939-4254-a633-a5f44e38b5bf","Type":"ContainerStarted","Data":"2de76707f84d5386e2f935be5659a25368ada001a0aed1edaa69f47350050d7a"}
Sep 16 12:08:38.854467 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.854437    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:38.938760 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:38.938734    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:39.427347 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:39.427320    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:39.818189 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:39.818155    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw" event={"ID":"11685ce0-7e31-48ff-b729-dfe81f8cfcde","Type":"ContainerStarted","Data":"39191c809143e19dd1d6a693eba7cefddb5ae938973fb876c78bd649f52d30b5"}
Sep 16 12:08:39.819802 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:39.819777    2558 generic.go:356] "Generic (PLEG): container finished" podID="f106348d-5939-4254-a633-a5f44e38b5bf" containerID="f51cbf4022bccf7247e765bdbc622e87436f174642c7576bc66798ee4bb5fd7f" exitCode=0
Sep 16 12:08:39.819923 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:39.819832    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vcpsg" event={"ID":"f106348d-5939-4254-a633-a5f44e38b5bf","Type":"ContainerDied","Data":"f51cbf4022bccf7247e765bdbc622e87436f174642c7576bc66798ee4bb5fd7f"}
Sep 16 12:08:39.839394 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:39.839345    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4cvlw" podStartSLOduration=1.6523391699999999 podStartE2EDuration="2.839331719s" totalImagesPullingTime="1.186992549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:39.838994511 +0000 UTC m=+316.592953243" watchObservedRunningTime="2026-09-16 12:08:39.839331719 +0000 UTC m=+316.593290450"
Sep 16 12:08:40.776242 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:40.776201    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:40.824770 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:40.824742    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vcpsg" event={"ID":"f106348d-5939-4254-a633-a5f44e38b5bf","Type":"ContainerStarted","Data":"7ee3b495e2558098637185ce895b17f8c0496fdf63c210853df2c0894c5457a1"}
Sep 16 12:08:40.825099 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:40.824795    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vcpsg" event={"ID":"f106348d-5939-4254-a633-a5f44e38b5bf","Type":"ContainerStarted","Data":"7e0d334f87d53a556feea2260818c5117bfc29c3187d96da83ea76967ac4f8de"}
Sep 16 12:08:40.847022 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:40.846984    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vcpsg" podStartSLOduration=2.570716801 podStartE2EDuration="3.84696963s" totalImagesPullingTime="1.207465872s" totalInitContainerRuntime="68.786957ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:40.845867971 +0000 UTC m=+317.599826702" watchObservedRunningTime="2026-09-16 12:08:40.84696963 +0000 UTC m=+317.600928360"
Sep 16 12:08:40.862420 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:40.862397    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:40.933532 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:40.933501    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:41.034006 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:41.033925    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:41.209135 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:41.209103    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:41.475471 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:41.475440    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:41.578388 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:41.578357    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:42.590438 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.590407    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz"]
Sep 16 12:08:42.590910 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.590892    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:42.593597 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.593582    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz"
Sep 16 12:08:42.595726 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.595703    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 12:08:42.595726 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.595705    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-t6xkf\""
Sep 16 12:08:42.600833 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.600812    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz"]
Sep 16 12:08:42.655456 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.655428    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:42.773086 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.773059    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6089f074-9576-4c3e-b3ae-7c59d85a1de4-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-42wdz\" (UID: \"6089f074-9576-4c3e-b3ae-7c59d85a1de4\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz"
Sep 16 12:08:42.773238 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.773124    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphrt\" (UniqueName: \"kubernetes.io/projected/6089f074-9576-4c3e-b3ae-7c59d85a1de4-kube-api-access-qphrt\") pod \"monitoring-plugin-5d85778b9d-42wdz\" (UID: \"6089f074-9576-4c3e-b3ae-7c59d85a1de4\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz"
Sep 16 12:08:42.874266 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.874233    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphrt\" (UniqueName: \"kubernetes.io/projected/6089f074-9576-4c3e-b3ae-7c59d85a1de4-kube-api-access-qphrt\") pod \"monitoring-plugin-5d85778b9d-42wdz\" (UID: \"6089f074-9576-4c3e-b3ae-7c59d85a1de4\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz"
Sep 16 12:08:42.874437 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.874278    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6089f074-9576-4c3e-b3ae-7c59d85a1de4-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-42wdz\" (UID: \"6089f074-9576-4c3e-b3ae-7c59d85a1de4\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz"
Sep 16 12:08:42.876904 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.876877    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6089f074-9576-4c3e-b3ae-7c59d85a1de4-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-42wdz\" (UID: \"6089f074-9576-4c3e-b3ae-7c59d85a1de4\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz"
Sep 16 12:08:42.881653 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.881633    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphrt\" (UniqueName: \"kubernetes.io/projected/6089f074-9576-4c3e-b3ae-7c59d85a1de4-kube-api-access-qphrt\") pod \"monitoring-plugin-5d85778b9d-42wdz\" (UID: \"6089f074-9576-4c3e-b3ae-7c59d85a1de4\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz"
Sep 16 12:08:42.903408 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.903382    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz"
Sep 16 12:08:42.951535 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:42.951500    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:43.070854 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.070823    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-658d964cd-d8l8g"]
Sep 16 12:08:43.075024 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.075000    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.077391 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.077367    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 12:08:43.077494 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.077440    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 12:08:43.077738 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.077723    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kd6bf\""
Sep 16 12:08:43.077860 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.077843    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 12:08:43.078405 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.078386    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 12:08:43.078490 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.078460    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 12:08:43.083134 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.083113    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 12:08:43.088895 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.088874    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-658d964cd-d8l8g"]
Sep 16 12:08:43.168258 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.168126    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz"]
Sep 16 12:08:43.170738 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:08:43.170704    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6089f074_9576_4c3e_b3ae_7c59d85a1de4.slice/crio-ae7d8ecf32b5fa0fcc53d33397e2f805e6fa5d1f8660329a63da04e332f63af7 WatchSource:0}: Error finding container ae7d8ecf32b5fa0fcc53d33397e2f805e6fa5d1f8660329a63da04e332f63af7: Status 404 returned error can't find the container with id ae7d8ecf32b5fa0fcc53d33397e2f805e6fa5d1f8660329a63da04e332f63af7
Sep 16 12:08:43.176365 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.176344    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-telemeter-client-tls\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.176485 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.176392    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-metrics-client-ca\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.176485 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.176419    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhr6j\" (UniqueName: \"kubernetes.io/projected/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-kube-api-access-nhr6j\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.176606 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.176485    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.176606 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.176520    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.176682 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.176611    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-serving-certs-ca-bundle\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.176682 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.176649    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-federate-client-tls\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.176682 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.176674    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-secret-telemeter-client\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.256059 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.256013    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:43.277146 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.277120    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.277237 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.277157    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-serving-certs-ca-bundle\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.277237 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.277181    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-federate-client-tls\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.277237 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.277200    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-secret-telemeter-client\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.277392 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.277236    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-telemeter-client-tls\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.277392 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.277267    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-metrics-client-ca\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.277392 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.277289    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhr6j\" (UniqueName: \"kubernetes.io/projected/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-kube-api-access-nhr6j\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.277392 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.277327    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.278084 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.278061    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-metrics-client-ca\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.278161 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.278063    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-serving-certs-ca-bundle\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.278425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.278401    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.279709 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.279686    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.279864 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.279845    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-telemeter-client-tls\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.280052 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.280036    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-federate-client-tls\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.280107 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.280092    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-secret-telemeter-client\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.285116 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.285099    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhr6j\" (UniqueName: \"kubernetes.io/projected/8b64bb41-3b66-43ff-9288-52dd90d7e6a1-kube-api-access-nhr6j\") pod \"telemeter-client-658d964cd-d8l8g\" (UID: \"8b64bb41-3b66-43ff-9288-52dd90d7e6a1\") " pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.356844 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.356808    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:43.384763 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.384736    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g"
Sep 16 12:08:43.661897 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.661866    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-658d964cd-d8l8g"]
Sep 16 12:08:43.665421 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:08:43.665398    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b64bb41_3b66_43ff_9288_52dd90d7e6a1.slice/crio-6fc8ead11dc535dcb85657af30a1ec3da39813daf86476cefc5e5ceda506f450 WatchSource:0}: Error finding container 6fc8ead11dc535dcb85657af30a1ec3da39813daf86476cefc5e5ceda506f450: Status 404 returned error can't find the container with id 6fc8ead11dc535dcb85657af30a1ec3da39813daf86476cefc5e5ceda506f450
Sep 16 12:08:43.835485 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.835405    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz" event={"ID":"6089f074-9576-4c3e-b3ae-7c59d85a1de4","Type":"ContainerStarted","Data":"ae7d8ecf32b5fa0fcc53d33397e2f805e6fa5d1f8660329a63da04e332f63af7"}
Sep 16 12:08:43.836696 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:43.836670    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g" event={"ID":"8b64bb41-3b66-43ff-9288-52dd90d7e6a1","Type":"ContainerStarted","Data":"6fc8ead11dc535dcb85657af30a1ec3da39813daf86476cefc5e5ceda506f450"}
Sep 16 12:08:44.084255 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:44.084222    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:44.346516 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:44.346488    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:44.410312 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:44.410282    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:44.842154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:44.842125    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz" event={"ID":"6089f074-9576-4c3e-b3ae-7c59d85a1de4","Type":"ContainerStarted","Data":"f9f7710cafe4c532e99ded8101cbe43e34c43a676f463020cb9cfc14a73c0e93"}
Sep 16 12:08:44.842582 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:44.842339    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz"
Sep 16 12:08:44.847560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:44.847525    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz"
Sep 16 12:08:44.857162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:44.857111    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-42wdz" podStartSLOduration=1.5372018170000001 podStartE2EDuration="2.857092158s" totalImagesPullingTime="1.319890341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:44.856008378 +0000 UTC m=+321.609967109" watchObservedRunningTime="2026-09-16 12:08:44.857092158 +0000 UTC m=+321.611050890"
Sep 16 12:08:45.772022 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:45.771995    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68fc5bf8bf-4d95z"
Sep 16 12:08:45.894165 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:45.894133    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:45.995232 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:45.995205    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:46.851045 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:46.851017    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g" event={"ID":"8b64bb41-3b66-43ff-9288-52dd90d7e6a1","Type":"ContainerStarted","Data":"478e973c29ae042331451dadb2b8ecbc1cc3ae64c7910a680b1ed44de05053d7"}
Sep 16 12:08:47.502691 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.502616    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66b86b5c44-p8wv5"]
Sep 16 12:08:47.505711 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.505695    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.508664 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.508640    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:08:47.508664 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.508663    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:08:47.508861 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.508673    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:08:47.508861 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.508673    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:08:47.508861 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.508748    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-982qg\""
Sep 16 12:08:47.508861 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.508758    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:08:47.509070 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.509054    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:08:47.509131 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.509079    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:08:47.512373 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.512352    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:08:47.517797 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.517774    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b86b5c44-p8wv5"]
Sep 16 12:08:47.613088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.613056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-console-config\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.613088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.613091    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjv9\" (UniqueName: \"kubernetes.io/projected/17e32744-591d-4ed3-965c-480174534832-kube-api-access-dxjv9\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.613283 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.613131    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17e32744-591d-4ed3-965c-480174534832-console-oauth-config\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.613283 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.613148    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-oauth-serving-cert\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.613283 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.613174    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-trusted-ca-bundle\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.613283 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.613192    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17e32744-591d-4ed3-965c-480174534832-console-serving-cert\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.613283 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.613210    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-service-ca\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.714601 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.714572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-console-config\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.714685 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.714625    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjv9\" (UniqueName: \"kubernetes.io/projected/17e32744-591d-4ed3-965c-480174534832-kube-api-access-dxjv9\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.714739 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.714688    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17e32744-591d-4ed3-965c-480174534832-console-oauth-config\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.714739 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.714715    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-oauth-serving-cert\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.714836 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.714764    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-trusted-ca-bundle\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.714836 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.714794    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17e32744-591d-4ed3-965c-480174534832-console-serving-cert\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.714836 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.714824    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-service-ca\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.715509 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.715459    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-console-config\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.715509 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.715478    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-service-ca\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.715509 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.715481    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-oauth-serving-cert\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.715736 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.715621    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-trusted-ca-bundle\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.717096 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.717077    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17e32744-591d-4ed3-965c-480174534832-console-oauth-config\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.717205 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.717192    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17e32744-591d-4ed3-965c-480174534832-console-serving-cert\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.722246 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.722221    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjv9\" (UniqueName: \"kubernetes.io/projected/17e32744-591d-4ed3-965c-480174534832-kube-api-access-dxjv9\") pod \"console-66b86b5c44-p8wv5\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") " pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.815993 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.815899    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:08:47.857024 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.856988    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g" event={"ID":"8b64bb41-3b66-43ff-9288-52dd90d7e6a1","Type":"ContainerStarted","Data":"6dbed054233802d4471ae7e0625ba3a45e8e3b5e02246c3d390a37ea6c13cbbb"}
Sep 16 12:08:47.857206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.857049    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g" event={"ID":"8b64bb41-3b66-43ff-9288-52dd90d7e6a1","Type":"ContainerStarted","Data":"a0e6f0a10643868bc05139b250efb367b333ad47ea1039ef674f218be3039fd2"}
Sep 16 12:08:47.883836 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:47.883762    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-658d964cd-d8l8g" podStartSLOduration=1.602717443 podStartE2EDuration="4.883739983s" totalImagesPullingTime="3.28102254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:47.882300042 +0000 UTC m=+324.636258772" watchObservedRunningTime="2026-09-16 12:08:47.883739983 +0000 UTC m=+324.637698715"
Sep 16 12:08:48.088152 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.088074    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b86b5c44-p8wv5"]
Sep 16 12:08:48.090882 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:08:48.090855    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e32744_591d_4ed3_965c_480174534832.slice/crio-8185b38828ec835857e2f9f01dc2053ceff4d0da8ae72c4ee823d4532f74428c WatchSource:0}: Error finding container 8185b38828ec835857e2f9f01dc2053ceff4d0da8ae72c4ee823d4532f74428c: Status 404 returned error can't find the container with id 8185b38828ec835857e2f9f01dc2053ceff4d0da8ae72c4ee823d4532f74428c
Sep 16 12:08:48.186854 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.186811    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8" podUID="6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" containerName="registry" containerID="cri-o://17f4d89f1e7d2de8ec7317cb1b2778fcf72a8d38ee9f20efdb21cbe4c3443a14" gracePeriod=30
Sep 16 12:08:48.419347 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.419324    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:08:48.520035 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.519997    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7f8m\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-kube-api-access-n7f8m\") pod \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") "
Sep 16 12:08:48.520035 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.520036    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-ca-trust-extracted\") pod \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") "
Sep 16 12:08:48.520564 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.520059    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-certificates\") pod \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") "
Sep 16 12:08:48.520564 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.520078    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-image-registry-private-configuration\") pod \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") "
Sep 16 12:08:48.520564 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.520102    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-installation-pull-secrets\") pod \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") "
Sep 16 12:08:48.520564 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.520528    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:08:48.522386 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.522363    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:08:48.522718 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.522691    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:08:48.522800 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.522726    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-kube-api-access-n7f8m" (OuterVolumeSpecName: "kube-api-access-n7f8m") pod "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01"). InnerVolumeSpecName "kube-api-access-n7f8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:08:48.527511 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.527491    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:08:48.533390 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.533361    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:48.592649 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.592628    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:48.620734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.620684    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-bound-sa-token\") pod \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") "
Sep 16 12:08:48.620734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.620712    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-trusted-ca\") pod \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") "
Sep 16 12:08:48.620838 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.620739    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") pod \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\" (UID: \"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01\") "
Sep 16 12:08:48.620922 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.620907    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7f8m\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-kube-api-access-n7f8m\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:08:48.620962 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.620932    2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-ca-trust-extracted\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:08:48.620962 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.620947    2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-certificates\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:08:48.621024 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.620961    2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-image-registry-private-configuration\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:08:48.621024 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.620974    2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-installation-pull-secrets\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:08:48.621093 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.621028    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:08:48.622651 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.622626    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:08:48.622738 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.622721    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" (UID: "6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:08:48.721298 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.721268    2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-bound-sa-token\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:08:48.721298 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.721292    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-trusted-ca\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:08:48.721298 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.721302    2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01-registry-tls\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:08:48.868733 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.868688    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b86b5c44-p8wv5" event={"ID":"17e32744-591d-4ed3-965c-480174534832","Type":"ContainerStarted","Data":"8185b38828ec835857e2f9f01dc2053ceff4d0da8ae72c4ee823d4532f74428c"}
Sep 16 12:08:48.869989 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.869967    2558 generic.go:356] "Generic (PLEG): container finished" podID="6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" containerID="17f4d89f1e7d2de8ec7317cb1b2778fcf72a8d38ee9f20efdb21cbe4c3443a14" exitCode=0
Sep 16 12:08:48.870113 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.870022    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8" event={"ID":"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01","Type":"ContainerDied","Data":"17f4d89f1e7d2de8ec7317cb1b2778fcf72a8d38ee9f20efdb21cbe4c3443a14"}
Sep 16 12:08:48.870113 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.870063    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8" event={"ID":"6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01","Type":"ContainerDied","Data":"e38095546a13b07c72292be72f0d819513b805228f91677e38887e76e18eafd7"}
Sep 16 12:08:48.870113 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.870064    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8785c58c-slgb8"
Sep 16 12:08:48.870113 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.870097    2558 scope.go:117] "RemoveContainer" containerID="17f4d89f1e7d2de8ec7317cb1b2778fcf72a8d38ee9f20efdb21cbe4c3443a14"
Sep 16 12:08:48.880130 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.880108    2558 scope.go:117] "RemoveContainer" containerID="17f4d89f1e7d2de8ec7317cb1b2778fcf72a8d38ee9f20efdb21cbe4c3443a14"
Sep 16 12:08:48.880436 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:08:48.880409    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f4d89f1e7d2de8ec7317cb1b2778fcf72a8d38ee9f20efdb21cbe4c3443a14\": container with ID starting with 17f4d89f1e7d2de8ec7317cb1b2778fcf72a8d38ee9f20efdb21cbe4c3443a14 not found: ID does not exist" containerID="17f4d89f1e7d2de8ec7317cb1b2778fcf72a8d38ee9f20efdb21cbe4c3443a14"
Sep 16 12:08:48.880520 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.880472    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f4d89f1e7d2de8ec7317cb1b2778fcf72a8d38ee9f20efdb21cbe4c3443a14"} err="failed to get container status \"17f4d89f1e7d2de8ec7317cb1b2778fcf72a8d38ee9f20efdb21cbe4c3443a14\": rpc error: code = NotFound desc = could not find container \"17f4d89f1e7d2de8ec7317cb1b2778fcf72a8d38ee9f20efdb21cbe4c3443a14\": container with ID starting with 17f4d89f1e7d2de8ec7317cb1b2778fcf72a8d38ee9f20efdb21cbe4c3443a14 not found: ID does not exist"
Sep 16 12:08:48.896980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.896953    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f8785c58c-slgb8"]
Sep 16 12:08:48.903342 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:48.903319    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f8785c58c-slgb8"]
Sep 16 12:08:49.757724 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:49.757695    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" path="/var/lib/kubelet/pods/6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01/volumes"
Sep 16 12:08:51.285141 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:51.285053    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66b86b5c44-p8wv5"]
Sep 16 12:08:51.481816 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:51.481780    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:51.584439 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:51.584350    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:51.883247 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:51.883212    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b86b5c44-p8wv5" event={"ID":"17e32744-591d-4ed3-965c-480174534832","Type":"ContainerStarted","Data":"7de0a3b4caab758ca18efa561d5c02853cc033bc7317ea7bb09c190ed25a6fc5"}
Sep 16 12:08:51.905631 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:51.905583    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66b86b5c44-p8wv5" podStartSLOduration=1.996903479 podStartE2EDuration="4.905568308s" totalImagesPullingTime="2.908664829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:51.904688646 +0000 UTC m=+328.658647377" watchObservedRunningTime="2026-09-16 12:08:51.905568308 +0000 UTC m=+328.659527331"
Sep 16 12:08:57.133001 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:57.132967    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:57.184285 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:57.184261    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:57.816566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:08:57.816526    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:09:02.199055 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:02.199017    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:02.297360 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:02.297327    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:02.398566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:02.398528    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:03.502359 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:03.502328    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:03.602942 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:03.602903    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:17.905031 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:17.904937    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66b86b5c44-p8wv5" podUID="17e32744-591d-4ed3-965c-480174534832" containerName="console" containerID="cri-o://7de0a3b4caab758ca18efa561d5c02853cc033bc7317ea7bb09c190ed25a6fc5" gracePeriod=15
Sep 16 12:09:18.143918 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.143894    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:09:18.242245 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.242159    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-console-config\") pod \"17e32744-591d-4ed3-965c-480174534832\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") "
Sep 16 12:09:18.242245 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.242214    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17e32744-591d-4ed3-965c-480174534832-console-oauth-config\") pod \"17e32744-591d-4ed3-965c-480174534832\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") "
Sep 16 12:09:18.242440 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.242247    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-trusted-ca-bundle\") pod \"17e32744-591d-4ed3-965c-480174534832\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") "
Sep 16 12:09:18.242440 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.242285    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17e32744-591d-4ed3-965c-480174534832-console-serving-cert\") pod \"17e32744-591d-4ed3-965c-480174534832\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") "
Sep 16 12:09:18.242440 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.242324    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxjv9\" (UniqueName: \"kubernetes.io/projected/17e32744-591d-4ed3-965c-480174534832-kube-api-access-dxjv9\") pod \"17e32744-591d-4ed3-965c-480174534832\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") "
Sep 16 12:09:18.242440 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.242350    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-service-ca\") pod \"17e32744-591d-4ed3-965c-480174534832\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") "
Sep 16 12:09:18.242440 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.242391    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-oauth-serving-cert\") pod \"17e32744-591d-4ed3-965c-480174534832\" (UID: \"17e32744-591d-4ed3-965c-480174534832\") "
Sep 16 12:09:18.242751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.242712    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-console-config" (OuterVolumeSpecName: "console-config") pod "17e32744-591d-4ed3-965c-480174534832" (UID: "17e32744-591d-4ed3-965c-480174534832"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:18.242884 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.242856    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "17e32744-591d-4ed3-965c-480174534832" (UID: "17e32744-591d-4ed3-965c-480174534832"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:18.242951 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.242874    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-service-ca" (OuterVolumeSpecName: "service-ca") pod "17e32744-591d-4ed3-965c-480174534832" (UID: "17e32744-591d-4ed3-965c-480174534832"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:18.242951 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.242855    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "17e32744-591d-4ed3-965c-480174534832" (UID: "17e32744-591d-4ed3-965c-480174534832"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:18.244593 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.244565    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e32744-591d-4ed3-965c-480174534832-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "17e32744-591d-4ed3-965c-480174534832" (UID: "17e32744-591d-4ed3-965c-480174534832"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:18.244685 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.244617    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e32744-591d-4ed3-965c-480174534832-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "17e32744-591d-4ed3-965c-480174534832" (UID: "17e32744-591d-4ed3-965c-480174534832"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:18.244774 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.244753    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e32744-591d-4ed3-965c-480174534832-kube-api-access-dxjv9" (OuterVolumeSpecName: "kube-api-access-dxjv9") pod "17e32744-591d-4ed3-965c-480174534832" (UID: "17e32744-591d-4ed3-965c-480174534832"). InnerVolumeSpecName "kube-api-access-dxjv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:09:18.342894 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.342859    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjv9\" (UniqueName: \"kubernetes.io/projected/17e32744-591d-4ed3-965c-480174534832-kube-api-access-dxjv9\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:09:18.342894 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.342889    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-service-ca\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:09:18.342894 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.342898    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-oauth-serving-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:09:18.343092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.342906    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-console-config\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:09:18.343092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.342915    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17e32744-591d-4ed3-965c-480174534832-console-oauth-config\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:09:18.343092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.342923    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e32744-591d-4ed3-965c-480174534832-trusted-ca-bundle\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:09:18.343092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.342931    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17e32744-591d-4ed3-965c-480174534832-console-serving-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:09:18.511750 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.511681    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:18.597213 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.597185    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:18.972246 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.972206    2558 generic.go:356] "Generic (PLEG): container finished" podID="17e32744-591d-4ed3-965c-480174534832" containerID="7de0a3b4caab758ca18efa561d5c02853cc033bc7317ea7bb09c190ed25a6fc5" exitCode=0
Sep 16 12:09:18.972669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.972254    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b86b5c44-p8wv5"
Sep 16 12:09:18.972669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.972293    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b86b5c44-p8wv5" event={"ID":"17e32744-591d-4ed3-965c-480174534832","Type":"ContainerDied","Data":"7de0a3b4caab758ca18efa561d5c02853cc033bc7317ea7bb09c190ed25a6fc5"}
Sep 16 12:09:18.972669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.972344    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b86b5c44-p8wv5" event={"ID":"17e32744-591d-4ed3-965c-480174534832","Type":"ContainerDied","Data":"8185b38828ec835857e2f9f01dc2053ceff4d0da8ae72c4ee823d4532f74428c"}
Sep 16 12:09:18.972669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.972364    2558 scope.go:117] "RemoveContainer" containerID="7de0a3b4caab758ca18efa561d5c02853cc033bc7317ea7bb09c190ed25a6fc5"
Sep 16 12:09:18.980505 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.980485    2558 scope.go:117] "RemoveContainer" containerID="7de0a3b4caab758ca18efa561d5c02853cc033bc7317ea7bb09c190ed25a6fc5"
Sep 16 12:09:18.980780 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:09:18.980761    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de0a3b4caab758ca18efa561d5c02853cc033bc7317ea7bb09c190ed25a6fc5\": container with ID starting with 7de0a3b4caab758ca18efa561d5c02853cc033bc7317ea7bb09c190ed25a6fc5 not found: ID does not exist" containerID="7de0a3b4caab758ca18efa561d5c02853cc033bc7317ea7bb09c190ed25a6fc5"
Sep 16 12:09:18.980844 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.980787    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de0a3b4caab758ca18efa561d5c02853cc033bc7317ea7bb09c190ed25a6fc5"} err="failed to get container status \"7de0a3b4caab758ca18efa561d5c02853cc033bc7317ea7bb09c190ed25a6fc5\": rpc error: code = NotFound desc = could not find container \"7de0a3b4caab758ca18efa561d5c02853cc033bc7317ea7bb09c190ed25a6fc5\": container with ID starting with 7de0a3b4caab758ca18efa561d5c02853cc033bc7317ea7bb09c190ed25a6fc5 not found: ID does not exist"
Sep 16 12:09:18.995105 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.995078    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66b86b5c44-p8wv5"]
Sep 16 12:09:18.998637 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:18.998615    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66b86b5c44-p8wv5"]
Sep 16 12:09:19.756540 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:19.756504    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17e32744-591d-4ed3-965c-480174534832" path="/var/lib/kubelet/pods/17e32744-591d-4ed3-965c-480174534832/volumes"
Sep 16 12:09:26.214833 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:26.214780    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:26.316368 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:26.316325    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:26.417532 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:26.417505    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:31.917019 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:31.916975    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj" podUID="60745e11-4f98-41fa-bf6b-8c9c58751509" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 12:09:37.494754 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:37.494716    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:09:37.497066 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:37.497041    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1617d1a8-1f68-4475-88f9-f6fdb91a07db-metrics-certs\") pod \"network-metrics-daemon-pm6d7\" (UID: \"1617d1a8-1f68-4475-88f9-f6fdb91a07db\") " pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:09:37.655823 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:37.655794    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kdnz\""
Sep 16 12:09:37.664332 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:37.664311    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pm6d7"
Sep 16 12:09:37.936184 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:37.936160    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pm6d7"]
Sep 16 12:09:37.938595 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:09:37.938538    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1617d1a8_1f68_4475_88f9_f6fdb91a07db.slice/crio-cf1e41aa56af3390063992ecd33615e6d7364ce39ec422732a898e3d279ced06 WatchSource:0}: Error finding container cf1e41aa56af3390063992ecd33615e6d7364ce39ec422732a898e3d279ced06: Status 404 returned error can't find the container with id cf1e41aa56af3390063992ecd33615e6d7364ce39ec422732a898e3d279ced06
Sep 16 12:09:38.033210 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:38.033176    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pm6d7" event={"ID":"1617d1a8-1f68-4475-88f9-f6fdb91a07db","Type":"ContainerStarted","Data":"cf1e41aa56af3390063992ecd33615e6d7364ce39ec422732a898e3d279ced06"}
Sep 16 12:09:39.039513 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:39.039474    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pm6d7" event={"ID":"1617d1a8-1f68-4475-88f9-f6fdb91a07db","Type":"ContainerStarted","Data":"4e76c496724232a4a01361b1ab4fefdafd3bbef3fef8be99cda27f6ee09a42af"}
Sep 16 12:09:39.656175 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:39.656144    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:39.724086 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:39.724056    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:40.047066 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:40.046964    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pm6d7" event={"ID":"1617d1a8-1f68-4475-88f9-f6fdb91a07db","Type":"ContainerStarted","Data":"667586eee36968f7423ebefc22cf01ccf830caddac0e10e0d276e3a8cb4e3e13"}
Sep 16 12:09:40.064658 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:40.064605    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pm6d7" podStartSLOduration=375.088591118 podStartE2EDuration="6m16.064538938s" totalImagesPullingTime="975.94782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:09:40.063668538 +0000 UTC m=+376.817627274" watchObservedRunningTime="2026-09-16 12:09:40.064538938 +0000 UTC m=+376.818497669"
Sep 16 12:09:40.157029 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:40.156987    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:40.227405 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:40.227363    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:41.916695 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:41.916653    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj" podUID="60745e11-4f98-41fa-bf6b-8c9c58751509" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 12:09:42.307362 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:42.307283    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:42.409162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:42.409129    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:51.916690 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:51.916646    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj" podUID="60745e11-4f98-41fa-bf6b-8c9c58751509" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 12:09:51.917271 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:51.916740    2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj"
Sep 16 12:09:51.917396 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:51.917373    2558 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"0479f3519cea923ca034fbb8bb15a50542d748a8423714d54cb714a4de564dd9"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 16 12:09:51.917457 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:51.917425    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj" podUID="60745e11-4f98-41fa-bf6b-8c9c58751509" containerName="service-proxy" containerID="cri-o://0479f3519cea923ca034fbb8bb15a50542d748a8423714d54cb714a4de564dd9" gracePeriod=30
Sep 16 12:09:52.088209 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:52.088158    2558 generic.go:356] "Generic (PLEG): container finished" podID="60745e11-4f98-41fa-bf6b-8c9c58751509" containerID="0479f3519cea923ca034fbb8bb15a50542d748a8423714d54cb714a4de564dd9" exitCode=2
Sep 16 12:09:52.088298 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:52.088222    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj" event={"ID":"60745e11-4f98-41fa-bf6b-8c9c58751509","Type":"ContainerDied","Data":"0479f3519cea923ca034fbb8bb15a50542d748a8423714d54cb714a4de564dd9"}
Sep 16 12:09:53.094013 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:53.093980    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f4869fb46-bvtfj" event={"ID":"60745e11-4f98-41fa-bf6b-8c9c58751509","Type":"ContainerStarted","Data":"727d6813f060cf2ffb6bec2c897ebcb74bfafe0861e4c8a1e639c7de5fd7b6db"}
Sep 16 12:09:53.952937 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:53.952890    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:54.017350 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:09:54.017319    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:01.827859 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.827824    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c9796558-jgsdg"]
Sep 16 12:10:01.828229 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.828075    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e32744-591d-4ed3-965c-480174534832" containerName="console"
Sep 16 12:10:01.828229 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.828088    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e32744-591d-4ed3-965c-480174534832" containerName="console"
Sep 16 12:10:01.828229 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.828098    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" containerName="registry"
Sep 16 12:10:01.828229 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.828104    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" containerName="registry"
Sep 16 12:10:01.828229 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.828161    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c9a48ef-d9c6-4ea9-ad86-6f1a9efaba01" containerName="registry"
Sep 16 12:10:01.828229 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.828169    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e32744-591d-4ed3-965c-480174534832" containerName="console"
Sep 16 12:10:01.832053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.832036    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.834446 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.834418    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-982qg\""
Sep 16 12:10:01.834446 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.834435    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:10:01.834645 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.834457    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:10:01.834735 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.834722    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:10:01.834788 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.834779    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:10:01.834843 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.834792    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:10:01.834914 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.834897    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:10:01.835163 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.835151    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:10:01.840974 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.840955    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:10:01.843733 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.843707    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c9796558-jgsdg"]
Sep 16 12:10:01.871848 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.871821    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-config\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.871967 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.871856    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-trusted-ca-bundle\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.871967 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.871872    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-oauth-serving-cert\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.871967 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.871912    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-oauth-config\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.872067 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.871965    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-service-ca\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.872067 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.871989    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-serving-cert\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.872067 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.872003    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2spmn\" (UniqueName: \"kubernetes.io/projected/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-kube-api-access-2spmn\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.890212 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.890188    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:01.973049 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.973014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-trusted-ca-bundle\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.973049 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.973048    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-oauth-serving-cert\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.973277 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.973072    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-oauth-config\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.973277 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.973111    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-service-ca\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.973277 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.973146    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-serving-cert\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.973277 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.973169    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2spmn\" (UniqueName: \"kubernetes.io/projected/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-kube-api-access-2spmn\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.973615 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.973588    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-config\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.975790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.974189    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-service-ca\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.975790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.974189    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-oauth-serving-cert\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.975790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.974352    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-trusted-ca-bundle\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.975790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.974477    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-config\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.978632 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.976354    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-oauth-config\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.978632 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.976507    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-serving-cert\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:01.982673 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:01.982655    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2spmn\" (UniqueName: \"kubernetes.io/projected/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-kube-api-access-2spmn\") pod \"console-c9796558-jgsdg\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") " pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:02.043589 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:02.043533    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:02.142166 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:02.142138    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:02.207652 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:02.207624    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:02.413386 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:02.413296    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c9796558-jgsdg"]
Sep 16 12:10:02.417604 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:10:02.417573    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d62dd6c_6e5e_4b54_8c6c_ad5f5bb941e5.slice/crio-d920d8e22b44b90357b7ea59748662f8be6daa59d5e50e1ef0807dce339542d7 WatchSource:0}: Error finding container d920d8e22b44b90357b7ea59748662f8be6daa59d5e50e1ef0807dce339542d7: Status 404 returned error can't find the container with id d920d8e22b44b90357b7ea59748662f8be6daa59d5e50e1ef0807dce339542d7
Sep 16 12:10:02.719151 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:02.719062    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:02.819610 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:02.819578    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:03.128447 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:03.128411    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c9796558-jgsdg" event={"ID":"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5","Type":"ContainerStarted","Data":"b83ffe9147461eb0d5a0d1eecce63ee3b8692468747850356652ee6b6d8038f6"}
Sep 16 12:10:03.128850 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:03.128468    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c9796558-jgsdg" event={"ID":"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5","Type":"ContainerStarted","Data":"d920d8e22b44b90357b7ea59748662f8be6daa59d5e50e1ef0807dce339542d7"}
Sep 16 12:10:03.145398 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:03.145349    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c9796558-jgsdg" podStartSLOduration=2.145336101 podStartE2EDuration="2.145336101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:10:03.144173 +0000 UTC m=+399.898131732" watchObservedRunningTime="2026-09-16 12:10:03.145336101 +0000 UTC m=+399.899294832"
Sep 16 12:10:05.823585 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:05.823532    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:05.924462 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:05.924427    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:11.805567 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:11.805525    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:11.862428 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:11.862398    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:12.142778 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:12.142751    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:12.142998 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:12.142981    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:12.147504 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:12.147482    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:12.164487 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:12.164458    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:10:12.309654 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:12.309623    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:12.411151 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:12.411060    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:12.553851 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:12.553820    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:18.516439 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:18.516410    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:18.616657 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:18.616624    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:59.037907 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:59.037824    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:59.089394 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:10:59.089362    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:08.107193 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:08.107162    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:08.204610 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:08.204576    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:15.906654 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:15.906595    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:23.944727 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:23.944695    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:28.628331 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:28.628302    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:29.656281 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:29.656248    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:31.471947 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:31.471916    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:33.302855 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:33.302822    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:33.409066 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:33.409038    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:35.514971 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:35.514935    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:35.615631 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:35.615598    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:36.178244 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:36.178213    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:36.280515 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:36.280487    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:37.519450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:37.519419    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:37.925318 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:37.925263    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:40.262917 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:40.262886    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:40.363194 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:40.363164    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:44.085720 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.085684    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dt4cn"]
Sep 16 12:11:44.088632 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.088615    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dt4cn"
Sep 16 12:11:44.090771 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.090747    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-88wbj\""
Sep 16 12:11:44.090771 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.090763    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 12:11:44.090941 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.090823    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 12:11:44.095834 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.095813    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dt4cn"]
Sep 16 12:11:44.103773 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.103753    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32d20c64-b4e8-45e8-b846-91e96a4048c9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dt4cn\" (UID: \"32d20c64-b4e8-45e8-b846-91e96a4048c9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dt4cn"
Sep 16 12:11:44.103896 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.103784    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gs46\" (UniqueName: \"kubernetes.io/projected/32d20c64-b4e8-45e8-b846-91e96a4048c9-kube-api-access-2gs46\") pod \"cert-manager-webhook-6486ffd997-dt4cn\" (UID: \"32d20c64-b4e8-45e8-b846-91e96a4048c9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dt4cn"
Sep 16 12:11:44.204571 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.204508    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32d20c64-b4e8-45e8-b846-91e96a4048c9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dt4cn\" (UID: \"32d20c64-b4e8-45e8-b846-91e96a4048c9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dt4cn"
Sep 16 12:11:44.204571 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.204573    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gs46\" (UniqueName: \"kubernetes.io/projected/32d20c64-b4e8-45e8-b846-91e96a4048c9-kube-api-access-2gs46\") pod \"cert-manager-webhook-6486ffd997-dt4cn\" (UID: \"32d20c64-b4e8-45e8-b846-91e96a4048c9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dt4cn"
Sep 16 12:11:44.213218 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.213195    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32d20c64-b4e8-45e8-b846-91e96a4048c9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dt4cn\" (UID: \"32d20c64-b4e8-45e8-b846-91e96a4048c9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dt4cn"
Sep 16 12:11:44.213341 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.213260    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gs46\" (UniqueName: \"kubernetes.io/projected/32d20c64-b4e8-45e8-b846-91e96a4048c9-kube-api-access-2gs46\") pod \"cert-manager-webhook-6486ffd997-dt4cn\" (UID: \"32d20c64-b4e8-45e8-b846-91e96a4048c9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dt4cn"
Sep 16 12:11:44.397992 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.397958    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dt4cn"
Sep 16 12:11:44.663700 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:44.663470    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dt4cn"]
Sep 16 12:11:44.666421 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:11:44.666384    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32d20c64_b4e8_45e8_b846_91e96a4048c9.slice/crio-07c05bb706adbfcf3fb5002914bef40e2f868ffce4d2ee38b47a8f79ebaa7fda WatchSource:0}: Error finding container 07c05bb706adbfcf3fb5002914bef40e2f868ffce4d2ee38b47a8f79ebaa7fda: Status 404 returned error can't find the container with id 07c05bb706adbfcf3fb5002914bef40e2f868ffce4d2ee38b47a8f79ebaa7fda
Sep 16 12:11:45.482865 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:45.482800    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dt4cn" event={"ID":"32d20c64-b4e8-45e8-b846-91e96a4048c9","Type":"ContainerStarted","Data":"07c05bb706adbfcf3fb5002914bef40e2f868ffce4d2ee38b47a8f79ebaa7fda"}
Sep 16 12:11:46.118509 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:46.118467    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:47.687221 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:47.687188    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:47.737834 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:47.737747    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:48.496865 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:48.496831    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dt4cn" event={"ID":"32d20c64-b4e8-45e8-b846-91e96a4048c9","Type":"ContainerStarted","Data":"14a385a9bd0bae343155a95dd6d8888851f718dacb7022ddfa5927d1efeabc83"}
Sep 16 12:11:48.497051 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:48.496912    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dt4cn"
Sep 16 12:11:48.511893 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:48.511848    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-dt4cn" podStartSLOduration=1.725579735 podStartE2EDuration="4.511832501s" totalImagesPullingTime="2.786252766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:11:48.51083887 +0000 UTC m=+505.264797613" watchObservedRunningTime="2026-09-16 12:11:48.511832501 +0000 UTC m=+505.265791231"
Sep 16 12:11:52.866186 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:52.866152    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:52.966713 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:52.966682    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:54.501966 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:54.501938    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dt4cn"
Sep 16 12:11:59.576102 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:59.576063    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:59.716703 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:11:59.716674    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:01.630781 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:01.630750    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:09.893984 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:09.893946    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:09.986586 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:09.986541    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:13.221089 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:13.221058    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:13.326916 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:13.326885    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:13.843241 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:13.843199    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:14.260560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.260474    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:14.371236 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.371203    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:14.389891 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.389866    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"]
Sep 16 12:12:14.396004 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.395987    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.398442 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.398422    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 12:12:14.398563 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.398531    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 12:12:14.398868 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.398843    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 12:12:14.398975 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.398955    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 12:12:14.399194 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.399178    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nps9s\""
Sep 16 12:12:14.423721 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.423694    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"]
Sep 16 12:12:14.540844 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.540755    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85c42150-374a-4c5c-8a55-d310175500f0-tmp\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.540844 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.540803    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smt22\" (UniqueName: \"kubernetes.io/projected/85c42150-374a-4c5c-8a55-d310175500f0-kube-api-access-smt22\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.540844 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.540833    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85c42150-374a-4c5c-8a55-d310175500f0-manifests\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.541060 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.540856    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85c42150-374a-4c5c-8a55-d310175500f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.541060 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.540877    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85c42150-374a-4c5c-8a55-d310175500f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.607077 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.607048    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:14.642184 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.642158    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85c42150-374a-4c5c-8a55-d310175500f0-tmp\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.642296 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.642194    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smt22\" (UniqueName: \"kubernetes.io/projected/85c42150-374a-4c5c-8a55-d310175500f0-kube-api-access-smt22\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.642296 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.642217    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85c42150-374a-4c5c-8a55-d310175500f0-manifests\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.642369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.642340    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85c42150-374a-4c5c-8a55-d310175500f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.642403 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.642367    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85c42150-374a-4c5c-8a55-d310175500f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.642597 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.642577    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85c42150-374a-4c5c-8a55-d310175500f0-manifests\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.644600 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.644580    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85c42150-374a-4c5c-8a55-d310175500f0-tmp\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.644725 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.644702    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85c42150-374a-4c5c-8a55-d310175500f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.644796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.644709    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85c42150-374a-4c5c-8a55-d310175500f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.650428 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.650407    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smt22\" (UniqueName: \"kubernetes.io/projected/85c42150-374a-4c5c-8a55-d310175500f0-kube-api-access-smt22\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5\" (UID: \"85c42150-374a-4c5c-8a55-d310175500f0\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.703245 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.703218    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:14.706230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.706206    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:14.804254 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.804175    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:14.906857 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.906819    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:14.997227 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:14.997194    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"]
Sep 16 12:12:15.001114 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:12:15.001081    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c42150_374a_4c5c_8a55_d310175500f0.slice/crio-4b242133561f6d67d4fac4c1c4eb084e50ed496c78a360bc87ca8a26304fcc7c WatchSource:0}: Error finding container 4b242133561f6d67d4fac4c1c4eb084e50ed496c78a360bc87ca8a26304fcc7c: Status 404 returned error can't find the container with id 4b242133561f6d67d4fac4c1c4eb084e50ed496c78a360bc87ca8a26304fcc7c
Sep 16 12:12:15.032174 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:15.032146    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:15.132155 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:15.132128    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:15.231162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:15.231136    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:15.596168 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:15.596129    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5" event={"ID":"85c42150-374a-4c5c-8a55-d310175500f0","Type":"ContainerStarted","Data":"4b242133561f6d67d4fac4c1c4eb084e50ed496c78a360bc87ca8a26304fcc7c"}
Sep 16 12:12:18.609936 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:18.609897    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5" event={"ID":"85c42150-374a-4c5c-8a55-d310175500f0","Type":"ContainerStarted","Data":"eefb5bff4d722ed970c535d77c7c2697c74905c6b0206d848452b9a51f1e688d"}
Sep 16 12:12:19.615306 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:19.615274    2558 generic.go:356] "Generic (PLEG): container finished" podID="85c42150-374a-4c5c-8a55-d310175500f0" containerID="eefb5bff4d722ed970c535d77c7c2697c74905c6b0206d848452b9a51f1e688d" exitCode=0
Sep 16 12:12:19.615306 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:19.615310    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5" event={"ID":"85c42150-374a-4c5c-8a55-d310175500f0","Type":"ContainerDied","Data":"eefb5bff4d722ed970c535d77c7c2697c74905c6b0206d848452b9a51f1e688d"}
Sep 16 12:12:20.620239 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:20.620201    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5" event={"ID":"85c42150-374a-4c5c-8a55-d310175500f0","Type":"ContainerStarted","Data":"b32a876f56ae00dcd888b71d32737df44a72e949ab6bcdc6e8973709803ee88f"}
Sep 16 12:12:20.620619 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:20.620275    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:20.639955 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:20.639900    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5" podStartSLOduration=2.370684168 podStartE2EDuration="6.639871932s" totalImagesPullingTime="3.2672732s" totalInitContainerRuntime="1.001914564s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:20.638977276 +0000 UTC m=+537.392936007" watchObservedRunningTime="2026-09-16 12:12:20.639871932 +0000 UTC m=+537.393830666"
Sep 16 12:12:31.626186 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:31.626154    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5"
Sep 16 12:12:33.539429 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:33.539395    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:33.639434 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:33.639404    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:44.139357 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.139324    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:44.186147 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.186119    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"]
Sep 16 12:12:44.195756 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.195737    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"
Sep 16 12:12:44.198780 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.198758    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 12:12:44.198868 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.198791    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 12:12:44.198926 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.198758    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 12:12:44.198926 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.198758    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 12:12:44.199049 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.199021    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-t28rh\""
Sep 16 12:12:44.199448 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.199427    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"]
Sep 16 12:12:44.266721 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.266692    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848a691d-ef56-405d-a77e-d22cb0c71c76-tls-certs\") pod \"kube-auth-proxy-6f67fb6dbc-hrwds\" (UID: \"848a691d-ef56-405d-a77e-d22cb0c71c76\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"
Sep 16 12:12:44.266842 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.266755    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/848a691d-ef56-405d-a77e-d22cb0c71c76-tmp\") pod \"kube-auth-proxy-6f67fb6dbc-hrwds\" (UID: \"848a691d-ef56-405d-a77e-d22cb0c71c76\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"
Sep 16 12:12:44.266842 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.266781    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9grqg\" (UniqueName: \"kubernetes.io/projected/848a691d-ef56-405d-a77e-d22cb0c71c76-kube-api-access-9grqg\") pod \"kube-auth-proxy-6f67fb6dbc-hrwds\" (UID: \"848a691d-ef56-405d-a77e-d22cb0c71c76\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"
Sep 16 12:12:44.367056 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.367029    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9grqg\" (UniqueName: \"kubernetes.io/projected/848a691d-ef56-405d-a77e-d22cb0c71c76-kube-api-access-9grqg\") pod \"kube-auth-proxy-6f67fb6dbc-hrwds\" (UID: \"848a691d-ef56-405d-a77e-d22cb0c71c76\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"
Sep 16 12:12:44.367161 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.367063    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848a691d-ef56-405d-a77e-d22cb0c71c76-tls-certs\") pod \"kube-auth-proxy-6f67fb6dbc-hrwds\" (UID: \"848a691d-ef56-405d-a77e-d22cb0c71c76\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"
Sep 16 12:12:44.367202 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:12:44.367160    2558 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 16 12:12:44.367240 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:12:44.367213    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848a691d-ef56-405d-a77e-d22cb0c71c76-tls-certs podName:848a691d-ef56-405d-a77e-d22cb0c71c76 nodeName:}" failed. No retries permitted until 2026-09-16 12:12:44.867197662 +0000 UTC m=+561.621156384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/848a691d-ef56-405d-a77e-d22cb0c71c76-tls-certs") pod "kube-auth-proxy-6f67fb6dbc-hrwds" (UID: "848a691d-ef56-405d-a77e-d22cb0c71c76") : secret "kube-auth-proxy-tls" not found
Sep 16 12:12:44.367289 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.367252    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/848a691d-ef56-405d-a77e-d22cb0c71c76-tmp\") pod \"kube-auth-proxy-6f67fb6dbc-hrwds\" (UID: \"848a691d-ef56-405d-a77e-d22cb0c71c76\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"
Sep 16 12:12:44.369329 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.369309    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/848a691d-ef56-405d-a77e-d22cb0c71c76-tmp\") pod \"kube-auth-proxy-6f67fb6dbc-hrwds\" (UID: \"848a691d-ef56-405d-a77e-d22cb0c71c76\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"
Sep 16 12:12:44.374823 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.374803    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9grqg\" (UniqueName: \"kubernetes.io/projected/848a691d-ef56-405d-a77e-d22cb0c71c76-kube-api-access-9grqg\") pod \"kube-auth-proxy-6f67fb6dbc-hrwds\" (UID: \"848a691d-ef56-405d-a77e-d22cb0c71c76\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"
Sep 16 12:12:44.528109 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.528032    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:44.677525 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.677492    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mwxr4"]
Sep 16 12:12:44.680759 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.680743    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4"
Sep 16 12:12:44.682829 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.682808    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 16 12:12:44.682946 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.682846    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-b2xgc\""
Sep 16 12:12:44.690490 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.690466    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mwxr4"]
Sep 16 12:12:44.713525 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.713493    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:44.770450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.770425    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbjxk\" (UniqueName: \"kubernetes.io/projected/ca95855b-0409-4f6c-b5e2-f6ad328dfadc-kube-api-access-cbjxk\") pod \"odh-model-controller-7998c67dc4-mwxr4\" (UID: \"ca95855b-0409-4f6c-b5e2-f6ad328dfadc\") " pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4"
Sep 16 12:12:44.770655 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.770628    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca95855b-0409-4f6c-b5e2-f6ad328dfadc-cert\") pod \"odh-model-controller-7998c67dc4-mwxr4\" (UID: \"ca95855b-0409-4f6c-b5e2-f6ad328dfadc\") " pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4"
Sep 16 12:12:44.787559 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.787474    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:44.871358 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.871326    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca95855b-0409-4f6c-b5e2-f6ad328dfadc-cert\") pod \"odh-model-controller-7998c67dc4-mwxr4\" (UID: \"ca95855b-0409-4f6c-b5e2-f6ad328dfadc\") " pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4"
Sep 16 12:12:44.871488 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.871377    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbjxk\" (UniqueName: \"kubernetes.io/projected/ca95855b-0409-4f6c-b5e2-f6ad328dfadc-kube-api-access-cbjxk\") pod \"odh-model-controller-7998c67dc4-mwxr4\" (UID: \"ca95855b-0409-4f6c-b5e2-f6ad328dfadc\") " pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4"
Sep 16 12:12:44.871576 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:12:44.871483    2558 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 12:12:44.871576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.871498    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848a691d-ef56-405d-a77e-d22cb0c71c76-tls-certs\") pod \"kube-auth-proxy-6f67fb6dbc-hrwds\" (UID: \"848a691d-ef56-405d-a77e-d22cb0c71c76\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"
Sep 16 12:12:44.871689 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:12:44.871607    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca95855b-0409-4f6c-b5e2-f6ad328dfadc-cert podName:ca95855b-0409-4f6c-b5e2-f6ad328dfadc nodeName:}" failed. No retries permitted until 2026-09-16 12:12:45.371585041 +0000 UTC m=+562.125543753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ca95855b-0409-4f6c-b5e2-f6ad328dfadc-cert") pod "odh-model-controller-7998c67dc4-mwxr4" (UID: "ca95855b-0409-4f6c-b5e2-f6ad328dfadc") : secret "odh-model-controller-webhook-cert" not found
Sep 16 12:12:44.873810 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.873785    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848a691d-ef56-405d-a77e-d22cb0c71c76-tls-certs\") pod \"kube-auth-proxy-6f67fb6dbc-hrwds\" (UID: \"848a691d-ef56-405d-a77e-d22cb0c71c76\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"
Sep 16 12:12:44.882033 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:44.882008    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbjxk\" (UniqueName: \"kubernetes.io/projected/ca95855b-0409-4f6c-b5e2-f6ad328dfadc-kube-api-access-cbjxk\") pod \"odh-model-controller-7998c67dc4-mwxr4\" (UID: \"ca95855b-0409-4f6c-b5e2-f6ad328dfadc\") " pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4"
Sep 16 12:12:45.007964 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.007933    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:45.089505 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.089423    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:45.105103 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.105085    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"
Sep 16 12:12:45.375660 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.375628    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"]
Sep 16 12:12:45.375998 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.375899    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca95855b-0409-4f6c-b5e2-f6ad328dfadc-cert\") pod \"odh-model-controller-7998c67dc4-mwxr4\" (UID: \"ca95855b-0409-4f6c-b5e2-f6ad328dfadc\") " pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4"
Sep 16 12:12:45.378417 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.378397    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca95855b-0409-4f6c-b5e2-f6ad328dfadc-cert\") pod \"odh-model-controller-7998c67dc4-mwxr4\" (UID: \"ca95855b-0409-4f6c-b5e2-f6ad328dfadc\") " pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4"
Sep 16 12:12:45.380082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.380063    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds"]
Sep 16 12:12:45.380169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.380147    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.381801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.381775    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:45.382196 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.382176    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 12:12:45.382305 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:12:45.382215    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848a691d_ef56_405d_a77e_d22cb0c71c76.slice/crio-51a246585afc6d5171dd201402cd36eaa64e99526d2a20df9320028acab0a832 WatchSource:0}: Error finding container 51a246585afc6d5171dd201402cd36eaa64e99526d2a20df9320028acab0a832: Status 404 returned error can't find the container with id 51a246585afc6d5171dd201402cd36eaa64e99526d2a20df9320028acab0a832
Sep 16 12:12:45.382305 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.382176    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 12:12:45.382305 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.382255    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 12:12:45.382305 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.382258    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 12:12:45.382846 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.382828    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5hbkg\""
Sep 16 12:12:45.393754 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.393724    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"]
Sep 16 12:12:45.458142 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.458117    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:45.476274 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.476252    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.476387 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.476278    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.476387 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.476301    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.476502 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.476390    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dnbh\" (UniqueName: \"kubernetes.io/projected/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-kube-api-access-2dnbh\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.476502 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.476429    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.476502 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.476459    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.476645 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.476504    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.519611 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.519578    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:45.576868 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.576842    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.576985 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.576893    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.576985 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.576910    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.577118 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.577099    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.577179 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.577164    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnbh\" (UniqueName: \"kubernetes.io/projected/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-kube-api-access-2dnbh\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.577237 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.577202    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.577286 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.577233    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.577760 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.577732    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.579195 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.579170    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.579443 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.579422    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.579485 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.579422    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.579575 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.579539    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.585445 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.585422    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.585531 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.585486    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dnbh\" (UniqueName: \"kubernetes.io/projected/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-kube-api-access-2dnbh\") pod \"istiod-openshift-gateway-7b7995d5cd-6dkqn\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.591258 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.591240    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4"
Sep 16 12:12:45.699084 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.699033    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:45.713214 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.713188    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds" event={"ID":"848a691d-ef56-405d-a77e-d22cb0c71c76","Type":"ContainerStarted","Data":"51a246585afc6d5171dd201402cd36eaa64e99526d2a20df9320028acab0a832"}
Sep 16 12:12:45.859652 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.859628    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mwxr4"]
Sep 16 12:12:45.862384 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:12:45.862360    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca95855b_0409_4f6c_b5e2_f6ad328dfadc.slice/crio-996833b1707db755b138770f21dd46377a69f1ab8fc7cd71b6c5ad06ef160c47 WatchSource:0}: Error finding container 996833b1707db755b138770f21dd46377a69f1ab8fc7cd71b6c5ad06ef160c47: Status 404 returned error can't find the container with id 996833b1707db755b138770f21dd46377a69f1ab8fc7cd71b6c5ad06ef160c47
Sep 16 12:12:45.975242 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:45.975204    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"]
Sep 16 12:12:45.978269 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:12:45.978239    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e6e795_e5fb_46a0_8fc5_a402759f0d5c.slice/crio-5ce4c33e9cf3c634b9ccd13de81439badc4b02de73f5c0f153c16d4495fc7cab WatchSource:0}: Error finding container 5ce4c33e9cf3c634b9ccd13de81439badc4b02de73f5c0f153c16d4495fc7cab: Status 404 returned error can't find the container with id 5ce4c33e9cf3c634b9ccd13de81439badc4b02de73f5c0f153c16d4495fc7cab
Sep 16 12:12:46.324322 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:46.324243    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:46.722501 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:46.722422    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn" event={"ID":"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c","Type":"ContainerStarted","Data":"5ce4c33e9cf3c634b9ccd13de81439badc4b02de73f5c0f153c16d4495fc7cab"}
Sep 16 12:12:46.726639 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:46.726610    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4" event={"ID":"ca95855b-0409-4f6c-b5e2-f6ad328dfadc","Type":"ContainerStarted","Data":"996833b1707db755b138770f21dd46377a69f1ab8fc7cd71b6c5ad06ef160c47"}
Sep 16 12:12:50.709724 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:50.709686    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:12:50.710010 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:50.709754    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:12:51.621026 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:51.620982    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:51.712330 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:51.712294    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:51.749418 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:51.749379    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn" event={"ID":"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c","Type":"ContainerStarted","Data":"3e39f2907a4dfca1dc6ed7fed217fc698d4db684564e80f5e74f2132b4fde7ea"}
Sep 16 12:12:51.749620 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:51.749584    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:51.751245 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:51.751207    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4" event={"ID":"ca95855b-0409-4f6c-b5e2-f6ad328dfadc","Type":"ContainerStarted","Data":"5a55339dbaca3563124217bf9e7072d492aa7a0a3c408fb8b2fb46eee13a168c"}
Sep 16 12:12:51.751385 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:51.751275    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4"
Sep 16 12:12:51.751385 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:51.751354    2558 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-6dkqn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:12:51.751501 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:51.751419    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn" podUID="b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:12:51.756368 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:51.756346    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds" event={"ID":"848a691d-ef56-405d-a77e-d22cb0c71c76","Type":"ContainerStarted","Data":"4d851be4b5156da6f46a97430be6a62d51fea943691985f6c60fc6e90d4e5924"}
Sep 16 12:12:51.768395 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:51.768355    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn" podStartSLOduration=2.039928169 podStartE2EDuration="6.768341943s" totalImagesPullingTime="4.728413774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:51.767333523 +0000 UTC m=+568.521292258" watchObservedRunningTime="2026-09-16 12:12:51.768341943 +0000 UTC m=+568.522300668"
Sep 16 12:12:51.782691 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:51.782651    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4" podStartSLOduration=2.936931209 podStartE2EDuration="7.782638451s" totalImagesPullingTime="4.845707242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:51.781964319 +0000 UTC m=+568.535923049" watchObservedRunningTime="2026-09-16 12:12:51.782638451 +0000 UTC m=+568.536597182"
Sep 16 12:12:51.799559 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:51.799513    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-hrwds" podStartSLOduration=2.486172235 podStartE2EDuration="7.799498738s" totalImagesPullingTime="5.313326503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:51.798476451 +0000 UTC m=+568.552435199" watchObservedRunningTime="2026-09-16 12:12:51.799498738 +0000 UTC m=+568.553457468"
Sep 16 12:12:52.022259 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.022183    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"]
Sep 16 12:12:52.025426 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.025409    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.038006 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.037984    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"]
Sep 16 12:12:52.131755 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.131720    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c521403f-3dad-48da-99ad-64300ec64ecd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.131914 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.131761    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5kcm\" (UniqueName: \"kubernetes.io/projected/c521403f-3dad-48da-99ad-64300ec64ecd-kube-api-access-g5kcm\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.131914 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.131800    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c521403f-3dad-48da-99ad-64300ec64ecd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.131914 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.131851    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c521403f-3dad-48da-99ad-64300ec64ecd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.132023 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.131927    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c521403f-3dad-48da-99ad-64300ec64ecd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.132023 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.131946    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c521403f-3dad-48da-99ad-64300ec64ecd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.132023 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.131978    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c521403f-3dad-48da-99ad-64300ec64ecd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.183137 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.183101    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:52.232570 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.232528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c521403f-3dad-48da-99ad-64300ec64ecd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.232686 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.232573    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5kcm\" (UniqueName: \"kubernetes.io/projected/c521403f-3dad-48da-99ad-64300ec64ecd-kube-api-access-g5kcm\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.232686 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.232609    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c521403f-3dad-48da-99ad-64300ec64ecd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.232686 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.232631    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c521403f-3dad-48da-99ad-64300ec64ecd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.232686 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.232658    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c521403f-3dad-48da-99ad-64300ec64ecd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.232686 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.232678    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c521403f-3dad-48da-99ad-64300ec64ecd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.232952 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.232704    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c521403f-3dad-48da-99ad-64300ec64ecd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.233432 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.233407    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c521403f-3dad-48da-99ad-64300ec64ecd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.234999 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.234974    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c521403f-3dad-48da-99ad-64300ec64ecd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.235093 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.234974    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c521403f-3dad-48da-99ad-64300ec64ecd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.235198 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.235178    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c521403f-3dad-48da-99ad-64300ec64ecd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.235369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.235349    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c521403f-3dad-48da-99ad-64300ec64ecd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.247086 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.247060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c521403f-3dad-48da-99ad-64300ec64ecd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.248813 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.248795    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5kcm\" (UniqueName: \"kubernetes.io/projected/c521403f-3dad-48da-99ad-64300ec64ecd-kube-api-access-g5kcm\") pod \"istiod-openshift-gateway-5677ccb49b-wtg7l\" (UID: \"c521403f-3dad-48da-99ad-64300ec64ecd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.334342 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.334256    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.389659 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.389609    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:52.636688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.636662    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"]
Sep 16 12:12:52.637990 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:12:52.637963    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc521403f_3dad_48da_99ad_64300ec64ecd.slice/crio-fd19572ef947ac71c6f437f1e38b889f5f33e43304613d04b2115a0b40c16064 WatchSource:0}: Error finding container fd19572ef947ac71c6f437f1e38b889f5f33e43304613d04b2115a0b40c16064: Status 404 returned error can't find the container with id fd19572ef947ac71c6f437f1e38b889f5f33e43304613d04b2115a0b40c16064
Sep 16 12:12:52.640855 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.640818    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:12:52.640912 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.640894    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:12:52.759776 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.759741    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l" event={"ID":"c521403f-3dad-48da-99ad-64300ec64ecd","Type":"ContainerStarted","Data":"e3c3fa4fe9267b6ba9ba7e5205ec9810f812887387f82a8eb31d613be0073fa9"}
Sep 16 12:12:52.760215 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.759803    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l" event={"ID":"c521403f-3dad-48da-99ad-64300ec64ecd","Type":"ContainerStarted","Data":"fd19572ef947ac71c6f437f1e38b889f5f33e43304613d04b2115a0b40c16064"}
Sep 16 12:12:52.760289 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.760220    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:52.760389 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.760368    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:52.782107 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.782064    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l" podStartSLOduration=0.782049137 podStartE2EDuration="782.049137ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:52.780490115 +0000 UTC m=+569.534448848" watchObservedRunningTime="2026-09-16 12:12:52.782049137 +0000 UTC m=+569.536007868"
Sep 16 12:12:52.789681 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:52.789660    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:53.767242 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:53.767214    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wtg7l"
Sep 16 12:12:53.830162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:53.830132    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"]
Sep 16 12:12:54.634448 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:54.634420    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:54.733406 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:54.733361    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:54.768710 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:54.768674    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn" podUID="b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" containerName="discovery" containerID="cri-o://3e39f2907a4dfca1dc6ed7fed217fc698d4db684564e80f5e74f2132b4fde7ea" gracePeriod=30
Sep 16 12:12:54.799154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:54.799130    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:55.011383 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.011354    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:55.056667 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.056635    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-kubeconfig\") pod \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") "
Sep 16 12:12:55.056858 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.056674    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-cacerts\") pod \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") "
Sep 16 12:12:55.056858 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.056711    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dnbh\" (UniqueName: \"kubernetes.io/projected/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-kube-api-access-2dnbh\") pod \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") "
Sep 16 12:12:55.056858 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.056781    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-csr-ca-configmap\") pod \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") "
Sep 16 12:12:55.056858 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.056811    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-csr-dns-cert\") pod \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") "
Sep 16 12:12:55.056858 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.056836    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-local-certs\") pod \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") "
Sep 16 12:12:55.056858 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.056859    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-token\") pod \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\" (UID: \"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c\") "
Sep 16 12:12:55.057282 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.057246    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" (UID: "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:12:55.060024 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.059992    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" (UID: "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:12:55.060134 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.060025    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-local-certs" (OuterVolumeSpecName: "local-certs") pod "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" (UID: "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:12:55.060282 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.060240    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-token" (OuterVolumeSpecName: "istio-token") pod "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" (UID: "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:12:55.060401 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.060329    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" (UID: "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:12:55.060401 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.060349    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-kube-api-access-2dnbh" (OuterVolumeSpecName: "kube-api-access-2dnbh") pod "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" (UID: "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c"). InnerVolumeSpecName "kube-api-access-2dnbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:12:55.061492 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.061467    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-cacerts" (OuterVolumeSpecName: "cacerts") pod "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" (UID: "b4e6e795-e5fb-46a0-8fc5-a402759f0d5c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:12:55.077684 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.077652    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:55.157860 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.157768    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-token\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:12:55.157860 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.157799    2558 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-kubeconfig\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:12:55.157860 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.157807    2558 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-cacerts\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:12:55.157860 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.157816    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dnbh\" (UniqueName: \"kubernetes.io/projected/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-kube-api-access-2dnbh\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:12:55.157860 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.157827    2558 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-csr-ca-configmap\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:12:55.157860 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.157836    2558 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-istio-csr-dns-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:12:55.157860 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.157844    2558 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c-local-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:12:55.599472 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.599445    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:55.686269 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.686241    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:55.774388 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.774355    2558 generic.go:356] "Generic (PLEG): container finished" podID="b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" containerID="3e39f2907a4dfca1dc6ed7fed217fc698d4db684564e80f5e74f2132b4fde7ea" exitCode=0
Sep 16 12:12:55.774790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.774398    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"
Sep 16 12:12:55.774790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.774433    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn" event={"ID":"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c","Type":"ContainerDied","Data":"3e39f2907a4dfca1dc6ed7fed217fc698d4db684564e80f5e74f2132b4fde7ea"}
Sep 16 12:12:55.774790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.774520    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn" event={"ID":"b4e6e795-e5fb-46a0-8fc5-a402759f0d5c","Type":"ContainerDied","Data":"5ce4c33e9cf3c634b9ccd13de81439badc4b02de73f5c0f153c16d4495fc7cab"}
Sep 16 12:12:55.774790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.774569    2558 scope.go:117] "RemoveContainer" containerID="3e39f2907a4dfca1dc6ed7fed217fc698d4db684564e80f5e74f2132b4fde7ea"
Sep 16 12:12:55.782742 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.782725    2558 scope.go:117] "RemoveContainer" containerID="3e39f2907a4dfca1dc6ed7fed217fc698d4db684564e80f5e74f2132b4fde7ea"
Sep 16 12:12:55.783006 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:12:55.782987    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e39f2907a4dfca1dc6ed7fed217fc698d4db684564e80f5e74f2132b4fde7ea\": container with ID starting with 3e39f2907a4dfca1dc6ed7fed217fc698d4db684564e80f5e74f2132b4fde7ea not found: ID does not exist" containerID="3e39f2907a4dfca1dc6ed7fed217fc698d4db684564e80f5e74f2132b4fde7ea"
Sep 16 12:12:55.783055 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.783014    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e39f2907a4dfca1dc6ed7fed217fc698d4db684564e80f5e74f2132b4fde7ea"} err="failed to get container status \"3e39f2907a4dfca1dc6ed7fed217fc698d4db684564e80f5e74f2132b4fde7ea\": rpc error: code = NotFound desc = could not find container \"3e39f2907a4dfca1dc6ed7fed217fc698d4db684564e80f5e74f2132b4fde7ea\": container with ID starting with 3e39f2907a4dfca1dc6ed7fed217fc698d4db684564e80f5e74f2132b4fde7ea not found: ID does not exist"
Sep 16 12:12:55.794379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.794351    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"]
Sep 16 12:12:55.798555 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.798522    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6dkqn"]
Sep 16 12:12:55.999062 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:55.998984    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:56.114644 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:56.114614    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:56.218074 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:56.218041    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:57.756959 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:12:57.756927    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" path="/var/lib/kubelet/pods/b4e6e795-e5fb-46a0-8fc5-a402759f0d5c/volumes"
Sep 16 12:13:02.761834 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:02.761801    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mwxr4"
Sep 16 12:13:04.873501 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:04.873468    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:04.932486 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:04.932459    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:41.304434 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.304396    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c5bf78c54-692zz"]
Sep 16 12:13:41.306045 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.304696    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" containerName="discovery"
Sep 16 12:13:41.306045 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.304709    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" containerName="discovery"
Sep 16 12:13:41.306045 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.304769    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e6e795-e5fb-46a0-8fc5-a402759f0d5c" containerName="discovery"
Sep 16 12:13:41.306464 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.306449    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.315757 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.315734    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab6b0364-809b-407a-b8e2-d949c6316add-console-oauth-config\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.315881 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.315767    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab6b0364-809b-407a-b8e2-d949c6316add-console-config\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.315881 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.315847    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6b0364-809b-407a-b8e2-d949c6316add-trusted-ca-bundle\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.315881 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.315877    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6b0364-809b-407a-b8e2-d949c6316add-service-ca\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.316020 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.315897    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab6b0364-809b-407a-b8e2-d949c6316add-oauth-serving-cert\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.316020 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.315916    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab6b0364-809b-407a-b8e2-d949c6316add-console-serving-cert\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.316020 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.315944    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lc85\" (UniqueName: \"kubernetes.io/projected/ab6b0364-809b-407a-b8e2-d949c6316add-kube-api-access-6lc85\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.319766 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.319743    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5bf78c54-692zz"]
Sep 16 12:13:41.364963 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.364933    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:41.416883 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.416855    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6b0364-809b-407a-b8e2-d949c6316add-trusted-ca-bundle\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.417032 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.416888    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6b0364-809b-407a-b8e2-d949c6316add-service-ca\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.417032 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.416906    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab6b0364-809b-407a-b8e2-d949c6316add-oauth-serving-cert\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.417032 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.416932    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab6b0364-809b-407a-b8e2-d949c6316add-console-serving-cert\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.417032 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.416958    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lc85\" (UniqueName: \"kubernetes.io/projected/ab6b0364-809b-407a-b8e2-d949c6316add-kube-api-access-6lc85\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.417032 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.416990    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab6b0364-809b-407a-b8e2-d949c6316add-console-oauth-config\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.417032 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.417011    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab6b0364-809b-407a-b8e2-d949c6316add-console-config\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.417721 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.417692    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab6b0364-809b-407a-b8e2-d949c6316add-oauth-serving-cert\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.417850 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.417749    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6b0364-809b-407a-b8e2-d949c6316add-service-ca\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.417850 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.417782    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab6b0364-809b-407a-b8e2-d949c6316add-console-config\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.417850 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.417842    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6b0364-809b-407a-b8e2-d949c6316add-trusted-ca-bundle\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.419561 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.419524    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab6b0364-809b-407a-b8e2-d949c6316add-console-oauth-config\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.419654 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.419524    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab6b0364-809b-407a-b8e2-d949c6316add-console-serving-cert\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.423871 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.423854    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lc85\" (UniqueName: \"kubernetes.io/projected/ab6b0364-809b-407a-b8e2-d949c6316add-kube-api-access-6lc85\") pod \"console-5c5bf78c54-692zz\" (UID: \"ab6b0364-809b-407a-b8e2-d949c6316add\") " pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.615627 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.615591    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:41.882985 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.882833    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5bf78c54-692zz"]
Sep 16 12:13:41.885641 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:13:41.885613    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6b0364_809b_407a_b8e2_d949c6316add.slice/crio-bd9befd731275d20242aaa935ff9703fe33370b3f60569630c603e24e22c14e7 WatchSource:0}: Error finding container bd9befd731275d20242aaa935ff9703fe33370b3f60569630c603e24e22c14e7: Status 404 returned error can't find the container with id bd9befd731275d20242aaa935ff9703fe33370b3f60569630c603e24e22c14e7
Sep 16 12:13:41.958498 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.958474    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5bf78c54-692zz" event={"ID":"ab6b0364-809b-407a-b8e2-d949c6316add","Type":"ContainerStarted","Data":"36fd9af303f77c212ec5403377ad29f6fa031e2ac7364c1eccd7015e91b4b862"}
Sep 16 12:13:41.958618 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:41.958527    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5bf78c54-692zz" event={"ID":"ab6b0364-809b-407a-b8e2-d949c6316add","Type":"ContainerStarted","Data":"bd9befd731275d20242aaa935ff9703fe33370b3f60569630c603e24e22c14e7"}
Sep 16 12:13:42.003491 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:42.003434    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c5bf78c54-692zz" podStartSLOduration=1.003416815 podStartE2EDuration="1.003416815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:13:42.002048053 +0000 UTC m=+618.756006784" watchObservedRunningTime="2026-09-16 12:13:42.003416815 +0000 UTC m=+618.757375548"
Sep 16 12:13:42.881084 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:42.881050    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:43.261655 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:43.261573    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:43.311168 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:43.311142    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:45.692347 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:45.692306    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:45.787977 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:45.787950    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:51.615975 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:51.615939    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:51.616384 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:51.616220    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:51.621009 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:51.620983    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:52.004148 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:52.004064    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c5bf78c54-692zz"
Sep 16 12:13:52.046966 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:52.046933    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c9796558-jgsdg"]
Sep 16 12:13:52.140463 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:52.140432    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:52.242218 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:52.242187    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:52.303954 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:52.303885    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:59.451780 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:13:59.451750    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:01.487712 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:01.487677    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"]
Sep 16 12:14:01.490916 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:01.490899    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:01.494455 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:01.494437    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 12:14:01.495221 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:01.495204    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9v667\""
Sep 16 12:14:01.495288 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:01.495242    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 12:14:01.504447 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:01.504427    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"]
Sep 16 12:14:01.577918 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:01.577885    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6fn\" (UniqueName: \"kubernetes.io/projected/b57b01bf-08fc-4f84-b58e-2a8c34b07cff-kube-api-access-pg6fn\") pod \"limitador-operator-controller-manager-74495c69f8-tjmf8\" (UID: \"b57b01bf-08fc-4f84-b58e-2a8c34b07cff\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:01.679364 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:01.679318    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6fn\" (UniqueName: \"kubernetes.io/projected/b57b01bf-08fc-4f84-b58e-2a8c34b07cff-kube-api-access-pg6fn\") pod \"limitador-operator-controller-manager-74495c69f8-tjmf8\" (UID: \"b57b01bf-08fc-4f84-b58e-2a8c34b07cff\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:01.689298 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:01.689269    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6fn\" (UniqueName: \"kubernetes.io/projected/b57b01bf-08fc-4f84-b58e-2a8c34b07cff-kube-api-access-pg6fn\") pod \"limitador-operator-controller-manager-74495c69f8-tjmf8\" (UID: \"b57b01bf-08fc-4f84-b58e-2a8c34b07cff\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:01.801048 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:01.800966    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:01.853474 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:01.853433    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:02.091011 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:02.090979    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"]
Sep 16 12:14:02.094636 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:14:02.094610    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57b01bf_08fc_4f84_b58e_2a8c34b07cff.slice/crio-7ba2e4a5e56a7049a669728acc3ac024c8583708d49faa5e3d84cee27fdb1f53 WatchSource:0}: Error finding container 7ba2e4a5e56a7049a669728acc3ac024c8583708d49faa5e3d84cee27fdb1f53: Status 404 returned error can't find the container with id 7ba2e4a5e56a7049a669728acc3ac024c8583708d49faa5e3d84cee27fdb1f53
Sep 16 12:14:02.096921 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:02.096900    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:14:02.251803 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:02.251757    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:03.057029 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:03.056998    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8" event={"ID":"b57b01bf-08fc-4f84-b58e-2a8c34b07cff","Type":"ContainerStarted","Data":"7ba2e4a5e56a7049a669728acc3ac024c8583708d49faa5e3d84cee27fdb1f53"}
Sep 16 12:14:03.410815 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:03.410778    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:03.969049 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:03.969013    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bwqqx"]
Sep 16 12:14:03.973360 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:03.973340    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-bwqqx"
Sep 16 12:14:03.977340 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:03.977323    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-n8j6r\""
Sep 16 12:14:03.983178 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:03.983159    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bwqqx"]
Sep 16 12:14:04.100359 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:04.100325    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wwl2\" (UniqueName: \"kubernetes.io/projected/53d0ab04-c567-4ee7-bfec-043856f64c9d-kube-api-access-8wwl2\") pod \"authorino-operator-76b478bd78-bwqqx\" (UID: \"53d0ab04-c567-4ee7-bfec-043856f64c9d\") " pod="kuadrant-system/authorino-operator-76b478bd78-bwqqx"
Sep 16 12:14:04.105588 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:04.105536    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:04.200979 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:04.200951    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wwl2\" (UniqueName: \"kubernetes.io/projected/53d0ab04-c567-4ee7-bfec-043856f64c9d-kube-api-access-8wwl2\") pod \"authorino-operator-76b478bd78-bwqqx\" (UID: \"53d0ab04-c567-4ee7-bfec-043856f64c9d\") " pod="kuadrant-system/authorino-operator-76b478bd78-bwqqx"
Sep 16 12:14:04.207591 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:04.207567    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:04.210413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:04.210392    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wwl2\" (UniqueName: \"kubernetes.io/projected/53d0ab04-c567-4ee7-bfec-043856f64c9d-kube-api-access-8wwl2\") pod \"authorino-operator-76b478bd78-bwqqx\" (UID: \"53d0ab04-c567-4ee7-bfec-043856f64c9d\") " pod="kuadrant-system/authorino-operator-76b478bd78-bwqqx"
Sep 16 12:14:04.284416 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:04.284337    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-bwqqx"
Sep 16 12:14:04.557031 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:04.557006    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bwqqx"]
Sep 16 12:14:04.559810 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:14:04.559785    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d0ab04_c567_4ee7_bfec_043856f64c9d.slice/crio-ac84e24fa416b9c5d94cb92c9244d3487f05bec793af58b236bc87d6a5e10c70 WatchSource:0}: Error finding container ac84e24fa416b9c5d94cb92c9244d3487f05bec793af58b236bc87d6a5e10c70: Status 404 returned error can't find the container with id ac84e24fa416b9c5d94cb92c9244d3487f05bec793af58b236bc87d6a5e10c70
Sep 16 12:14:04.645369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:04.645343    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:05.066440 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:05.066409    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-bwqqx" event={"ID":"53d0ab04-c567-4ee7-bfec-043856f64c9d","Type":"ContainerStarted","Data":"ac84e24fa416b9c5d94cb92c9244d3487f05bec793af58b236bc87d6a5e10c70"}
Sep 16 12:14:05.070951 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:05.070922    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:05.441009 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:05.440970    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:06.163607 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:06.163567    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:07.076896 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:07.076864    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-bwqqx" event={"ID":"53d0ab04-c567-4ee7-bfec-043856f64c9d","Type":"ContainerStarted","Data":"b11294b87bcd98feee1b3573158dd8e600138d459895baa571a0147337ea1cd2"}
Sep 16 12:14:07.077277 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:07.076935    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-bwqqx"
Sep 16 12:14:07.096996 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:07.096951    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-bwqqx" podStartSLOduration=1.672424615 podStartE2EDuration="4.096937812s" totalImagesPullingTime="2.424513197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:07.095881626 +0000 UTC m=+643.849840567" watchObservedRunningTime="2026-09-16 12:14:07.096937812 +0000 UTC m=+643.850896542"
Sep 16 12:14:07.113350 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:07.113326    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:07.190879 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:07.190849    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:08.859580 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:08.859530    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:09.079095 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:09.079063    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:10.512914 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:10.512876    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-krfqd"]
Sep 16 12:14:10.516205 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:10.516188    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-krfqd"
Sep 16 12:14:10.518336 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:10.518313    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2r9vv\""
Sep 16 12:14:10.518444 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:10.518316    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 12:14:10.532786 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:10.532763    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-krfqd"]
Sep 16 12:14:10.656813 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:10.656772    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv9hc\" (UniqueName: \"kubernetes.io/projected/7751cdd2-68e7-45e2-8f2a-32e90fc8559a-kube-api-access-wv9hc\") pod \"dns-operator-controller-manager-7b84cd7c9c-krfqd\" (UID: \"7751cdd2-68e7-45e2-8f2a-32e90fc8559a\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-krfqd"
Sep 16 12:14:10.757588 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:10.757525    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv9hc\" (UniqueName: \"kubernetes.io/projected/7751cdd2-68e7-45e2-8f2a-32e90fc8559a-kube-api-access-wv9hc\") pod \"dns-operator-controller-manager-7b84cd7c9c-krfqd\" (UID: \"7751cdd2-68e7-45e2-8f2a-32e90fc8559a\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-krfqd"
Sep 16 12:14:10.764980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:10.764934    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv9hc\" (UniqueName: \"kubernetes.io/projected/7751cdd2-68e7-45e2-8f2a-32e90fc8559a-kube-api-access-wv9hc\") pod \"dns-operator-controller-manager-7b84cd7c9c-krfqd\" (UID: \"7751cdd2-68e7-45e2-8f2a-32e90fc8559a\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-krfqd"
Sep 16 12:14:10.825625 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:10.825592    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-krfqd"
Sep 16 12:14:11.094230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:11.094204    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-krfqd"]
Sep 16 12:14:11.096577 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:14:11.096533    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751cdd2_68e7_45e2_8f2a_32e90fc8559a.slice/crio-29d940a3130d04d3dedb4b96903558553f00d93fb2bb194559633a6536311726 WatchSource:0}: Error finding container 29d940a3130d04d3dedb4b96903558553f00d93fb2bb194559633a6536311726: Status 404 returned error can't find the container with id 29d940a3130d04d3dedb4b96903558553f00d93fb2bb194559633a6536311726
Sep 16 12:14:11.858797 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:11.858766    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:11.960981 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:11.960949    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:12.100667 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:12.100624    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-krfqd" event={"ID":"7751cdd2-68e7-45e2-8f2a-32e90fc8559a","Type":"ContainerStarted","Data":"29d940a3130d04d3dedb4b96903558553f00d93fb2bb194559633a6536311726"}
Sep 16 12:14:13.106139 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:13.106108    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-krfqd" event={"ID":"7751cdd2-68e7-45e2-8f2a-32e90fc8559a","Type":"ContainerStarted","Data":"ade35e856349062fbdf4b77172cbd005a226a4226957292e12f4670f671ef6b3"}
Sep 16 12:14:13.106508 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:13.106231    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-krfqd"
Sep 16 12:14:13.122376 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:13.122329    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-krfqd" podStartSLOduration=1.5595752059999999 podStartE2EDuration="3.122314856s" totalImagesPullingTime="1.56273965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:13.120929982 +0000 UTC m=+649.874888713" watchObservedRunningTime="2026-09-16 12:14:13.122314856 +0000 UTC m=+649.876273587"
Sep 16 12:14:15.117430 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:15.117395    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8" event={"ID":"b57b01bf-08fc-4f84-b58e-2a8c34b07cff","Type":"ContainerStarted","Data":"2435fdd00d38796a8e2fd6946b516f82660118e263f7d2ca9157d20a719ac818"}
Sep 16 12:14:15.117817 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:15.117531    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:15.133541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:15.133494    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8" podStartSLOduration=1.495177784 podStartE2EDuration="14.133478094s" totalImagesPullingTime="12.63830031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:15.132430731 +0000 UTC m=+651.886389461" watchObservedRunningTime="2026-09-16 12:14:15.133478094 +0000 UTC m=+651.887436829"
Sep 16 12:14:15.843968 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:15.843938    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:15.944738 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:15.944698    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:16.044752 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:16.044716    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:16.107177 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:16.107102    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:17.066404 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.066362    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c9796558-jgsdg" podUID="5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5" containerName="console" containerID="cri-o://b83ffe9147461eb0d5a0d1eecce63ee3b8692468747850356652ee6b6d8038f6" gracePeriod=15
Sep 16 12:14:17.310100 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.310074    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:14:17.418971 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.418914    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-trusted-ca-bundle\") pod \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") "
Sep 16 12:14:17.418971 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.418974    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-service-ca\") pod \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") "
Sep 16 12:14:17.419201 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.419010    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-oauth-config\") pod \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") "
Sep 16 12:14:17.419201 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.419061    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-config\") pod \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") "
Sep 16 12:14:17.419201 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.419109    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-serving-cert\") pod \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") "
Sep 16 12:14:17.419201 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.419150    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-oauth-serving-cert\") pod \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") "
Sep 16 12:14:17.419201 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.419175    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2spmn\" (UniqueName: \"kubernetes.io/projected/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-kube-api-access-2spmn\") pod \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\" (UID: \"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5\") "
Sep 16 12:14:17.419456 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.419432    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5" (UID: "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:14:17.419516 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.419471    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-service-ca" (OuterVolumeSpecName: "service-ca") pod "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5" (UID: "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:14:17.419604 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.419585    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-config" (OuterVolumeSpecName: "console-config") pod "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5" (UID: "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:14:17.419673 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.419597    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5" (UID: "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:14:17.421356 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.421331    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5" (UID: "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:14:17.421467 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.421364    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-kube-api-access-2spmn" (OuterVolumeSpecName: "kube-api-access-2spmn") pod "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5" (UID: "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5"). InnerVolumeSpecName "kube-api-access-2spmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:14:17.421467 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.421367    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5" (UID: "5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:14:17.519766 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.519730    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-trusted-ca-bundle\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:14:17.519766 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.519761    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-service-ca\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:14:17.519766 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.519770    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-oauth-config\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:14:17.519986 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.519781    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-config\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:14:17.519986 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.519790    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-console-serving-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:14:17.519986 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.519798    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-oauth-serving-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:14:17.519986 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:17.519807    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2spmn\" (UniqueName: \"kubernetes.io/projected/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5-kube-api-access-2spmn\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:14:18.083489 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:18.083403    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-bwqqx"
Sep 16 12:14:18.132158 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:18.132125    2558 generic.go:356] "Generic (PLEG): container finished" podID="5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5" containerID="b83ffe9147461eb0d5a0d1eecce63ee3b8692468747850356652ee6b6d8038f6" exitCode=0
Sep 16 12:14:18.132316 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:18.132224    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c9796558-jgsdg"
Sep 16 12:14:18.132316 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:18.132223    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c9796558-jgsdg" event={"ID":"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5","Type":"ContainerDied","Data":"b83ffe9147461eb0d5a0d1eecce63ee3b8692468747850356652ee6b6d8038f6"}
Sep 16 12:14:18.132388 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:18.132348    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c9796558-jgsdg" event={"ID":"5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5","Type":"ContainerDied","Data":"d920d8e22b44b90357b7ea59748662f8be6daa59d5e50e1ef0807dce339542d7"}
Sep 16 12:14:18.132388 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:18.132371    2558 scope.go:117] "RemoveContainer" containerID="b83ffe9147461eb0d5a0d1eecce63ee3b8692468747850356652ee6b6d8038f6"
Sep 16 12:14:18.141024 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:18.141001    2558 scope.go:117] "RemoveContainer" containerID="b83ffe9147461eb0d5a0d1eecce63ee3b8692468747850356652ee6b6d8038f6"
Sep 16 12:14:18.141288 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:14:18.141271    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83ffe9147461eb0d5a0d1eecce63ee3b8692468747850356652ee6b6d8038f6\": container with ID starting with b83ffe9147461eb0d5a0d1eecce63ee3b8692468747850356652ee6b6d8038f6 not found: ID does not exist" containerID="b83ffe9147461eb0d5a0d1eecce63ee3b8692468747850356652ee6b6d8038f6"
Sep 16 12:14:18.141353 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:18.141298    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83ffe9147461eb0d5a0d1eecce63ee3b8692468747850356652ee6b6d8038f6"} err="failed to get container status \"b83ffe9147461eb0d5a0d1eecce63ee3b8692468747850356652ee6b6d8038f6\": rpc error: code = NotFound desc = could not find container \"b83ffe9147461eb0d5a0d1eecce63ee3b8692468747850356652ee6b6d8038f6\": container with ID starting with b83ffe9147461eb0d5a0d1eecce63ee3b8692468747850356652ee6b6d8038f6 not found: ID does not exist"
Sep 16 12:14:18.148035 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:18.148012    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c9796558-jgsdg"]
Sep 16 12:14:18.152269 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:18.152247    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c9796558-jgsdg"]
Sep 16 12:14:19.761715 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:19.761682    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5" path="/var/lib/kubelet/pods/5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5/volumes"
Sep 16 12:14:21.044364 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:21.044326    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:21.144578 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:21.144529    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:21.244429 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:21.244393    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:24.114466 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:24.114438    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-krfqd"
Sep 16 12:14:26.124845 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:26.124809    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:27.528783 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:27.528748    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:27.627717 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:27.627681    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:31.889286 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:31.889246    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:31.993486 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:31.993449    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:35.692140 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:35.692110    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:35.745719 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:35.745687    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:35.879380 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:35.879343    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:35.983261 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:35.983183    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:38.994491 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:38.994459    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:39.094534 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:39.094504    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:41.282293 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:41.282257    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:41.383813 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:41.383782    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:41.483443 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:41.483402    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:41.584503 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:41.584436    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:41.684657 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:41.684629    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:41.784992 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:41.784961    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:47.550706 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:47.550676    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:47.651613 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:47.651571    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:48.837097 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:48.837064    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"]
Sep 16 12:14:48.837490 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:48.837479    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5" containerName="console"
Sep 16 12:14:48.837580 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:48.837496    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5" containerName="console"
Sep 16 12:14:48.837652 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:48.837596    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d62dd6c-6e5e-4b54-8c6c-ad5f5bb941e5" containerName="console"
Sep 16 12:14:48.840415 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:48.840395    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:48.842704 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:48.842686    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mbkvc\""
Sep 16 12:14:48.851300 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:48.851279    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"]
Sep 16 12:14:48.874840 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:48.874808    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12a3dc10-068e-4d80-aa2d-3735e3bc3a01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kswdg\" (UID: \"12a3dc10-068e-4d80-aa2d-3735e3bc3a01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:48.874999 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:48.874877    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd9dx\" (UniqueName: \"kubernetes.io/projected/12a3dc10-068e-4d80-aa2d-3735e3bc3a01-kube-api-access-zd9dx\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kswdg\" (UID: \"12a3dc10-068e-4d80-aa2d-3735e3bc3a01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:48.976186 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:48.976148    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12a3dc10-068e-4d80-aa2d-3735e3bc3a01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kswdg\" (UID: \"12a3dc10-068e-4d80-aa2d-3735e3bc3a01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:48.976374 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:48.976214    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9dx\" (UniqueName: \"kubernetes.io/projected/12a3dc10-068e-4d80-aa2d-3735e3bc3a01-kube-api-access-zd9dx\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kswdg\" (UID: \"12a3dc10-068e-4d80-aa2d-3735e3bc3a01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:48.976642 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:48.976619    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12a3dc10-068e-4d80-aa2d-3735e3bc3a01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kswdg\" (UID: \"12a3dc10-068e-4d80-aa2d-3735e3bc3a01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:48.990685 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:48.990651    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd9dx\" (UniqueName: \"kubernetes.io/projected/12a3dc10-068e-4d80-aa2d-3735e3bc3a01-kube-api-access-zd9dx\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kswdg\" (UID: \"12a3dc10-068e-4d80-aa2d-3735e3bc3a01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:49.150393 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:49.150358    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:49.426484 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:49.426461    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"]
Sep 16 12:14:49.428521 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:14:49.428494    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a3dc10_068e_4d80_aa2d_3735e3bc3a01.slice/crio-62fc6d4f5e8dfb2c76c44bfda094199022d90cc4323742235be5cfdc17a08b6e WatchSource:0}: Error finding container 62fc6d4f5e8dfb2c76c44bfda094199022d90cc4323742235be5cfdc17a08b6e: Status 404 returned error can't find the container with id 62fc6d4f5e8dfb2c76c44bfda094199022d90cc4323742235be5cfdc17a08b6e
Sep 16 12:14:49.962587 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:49.962540    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"]
Sep 16 12:14:49.965888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:49.965856    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"]
Sep 16 12:14:49.986683 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:49.986652    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"]
Sep 16 12:14:49.991512 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:49.991491    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"
Sep 16 12:14:49.997947 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:49.997921    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"]
Sep 16 12:14:49.998292 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:49.998259    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8" podUID="b57b01bf-08fc-4f84-b58e-2a8c34b07cff" containerName="manager" containerID="cri-o://2435fdd00d38796a8e2fd6946b516f82660118e263f7d2ca9157d20a719ac818" gracePeriod=2
Sep 16 12:14:50.006735 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.006709    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"]
Sep 16 12:14:50.010368 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.010346    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"]
Sep 16 12:14:50.027041 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.027018    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-sg26x"]
Sep 16 12:14:50.027309 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.027297    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b57b01bf-08fc-4f84-b58e-2a8c34b07cff" containerName="manager"
Sep 16 12:14:50.027354 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.027312    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b57b01bf-08fc-4f84-b58e-2a8c34b07cff" containerName="manager"
Sep 16 12:14:50.027387 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.027371    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b57b01bf-08fc-4f84-b58e-2a8c34b07cff" containerName="manager"
Sep 16 12:14:50.033056 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.031061    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sg26x"
Sep 16 12:14:50.044197 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.044173    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-sg26x"]
Sep 16 12:14:50.064792 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:14:50.064751    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-tjmf8\" is forbidden: User \"system:node:ip-10-0-136-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-98.ec2.internal' and this object" podUID="b57b01bf-08fc-4f84-b58e-2a8c34b07cff" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:50.085908 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.085882    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a75d5e-6436-4b60-a85c-0ebd14ae5692-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-sxd4s\" (UID: \"b4a75d5e-6436-4b60-a85c-0ebd14ae5692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"
Sep 16 12:14:50.085999 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.085924    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfbkq\" (UniqueName: \"kubernetes.io/projected/b4a75d5e-6436-4b60-a85c-0ebd14ae5692-kube-api-access-sfbkq\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-sxd4s\" (UID: \"b4a75d5e-6436-4b60-a85c-0ebd14ae5692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"
Sep 16 12:14:50.085999 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.085979    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wztcj\" (UniqueName: \"kubernetes.io/projected/3b467b85-e56c-47dc-a9c5-42b2fed0f04d-kube-api-access-wztcj\") pod \"limitador-operator-controller-manager-74495c69f8-sg26x\" (UID: \"3b467b85-e56c-47dc-a9c5-42b2fed0f04d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sg26x"
Sep 16 12:14:50.187191 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.187154    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a75d5e-6436-4b60-a85c-0ebd14ae5692-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-sxd4s\" (UID: \"b4a75d5e-6436-4b60-a85c-0ebd14ae5692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"
Sep 16 12:14:50.187311 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.187230    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfbkq\" (UniqueName: \"kubernetes.io/projected/b4a75d5e-6436-4b60-a85c-0ebd14ae5692-kube-api-access-sfbkq\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-sxd4s\" (UID: \"b4a75d5e-6436-4b60-a85c-0ebd14ae5692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"
Sep 16 12:14:50.187311 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.187284    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wztcj\" (UniqueName: \"kubernetes.io/projected/3b467b85-e56c-47dc-a9c5-42b2fed0f04d-kube-api-access-wztcj\") pod \"limitador-operator-controller-manager-74495c69f8-sg26x\" (UID: \"3b467b85-e56c-47dc-a9c5-42b2fed0f04d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sg26x"
Sep 16 12:14:50.187527 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.187507    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a75d5e-6436-4b60-a85c-0ebd14ae5692-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-sxd4s\" (UID: \"b4a75d5e-6436-4b60-a85c-0ebd14ae5692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"
Sep 16 12:14:50.195373 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.195350    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wztcj\" (UniqueName: \"kubernetes.io/projected/3b467b85-e56c-47dc-a9c5-42b2fed0f04d-kube-api-access-wztcj\") pod \"limitador-operator-controller-manager-74495c69f8-sg26x\" (UID: \"3b467b85-e56c-47dc-a9c5-42b2fed0f04d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sg26x"
Sep 16 12:14:50.195611 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.195588    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfbkq\" (UniqueName: \"kubernetes.io/projected/b4a75d5e-6436-4b60-a85c-0ebd14ae5692-kube-api-access-sfbkq\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-sxd4s\" (UID: \"b4a75d5e-6436-4b60-a85c-0ebd14ae5692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"
Sep 16 12:14:50.239751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.239732    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:50.241531 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:14:50.241503    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-tjmf8\" is forbidden: User \"system:node:ip-10-0-136-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-98.ec2.internal' and this object" podUID="b57b01bf-08fc-4f84-b58e-2a8c34b07cff" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:50.267702 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.267674    2558 generic.go:356] "Generic (PLEG): container finished" podID="b57b01bf-08fc-4f84-b58e-2a8c34b07cff" containerID="2435fdd00d38796a8e2fd6946b516f82660118e263f7d2ca9157d20a719ac818" exitCode=0
Sep 16 12:14:50.267799 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.267716    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:50.267799 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.267719    2558 scope.go:117] "RemoveContainer" containerID="2435fdd00d38796a8e2fd6946b516f82660118e263f7d2ca9157d20a719ac818"
Sep 16 12:14:50.269959 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:14:50.269931    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-tjmf8\" is forbidden: User \"system:node:ip-10-0-136-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-98.ec2.internal' and this object" podUID="b57b01bf-08fc-4f84-b58e-2a8c34b07cff" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:50.275278 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.275264    2558 scope.go:117] "RemoveContainer" containerID="2435fdd00d38796a8e2fd6946b516f82660118e263f7d2ca9157d20a719ac818"
Sep 16 12:14:50.275535 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:14:50.275515    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2435fdd00d38796a8e2fd6946b516f82660118e263f7d2ca9157d20a719ac818\": container with ID starting with 2435fdd00d38796a8e2fd6946b516f82660118e263f7d2ca9157d20a719ac818 not found: ID does not exist" containerID="2435fdd00d38796a8e2fd6946b516f82660118e263f7d2ca9157d20a719ac818"
Sep 16 12:14:50.275604 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.275562    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2435fdd00d38796a8e2fd6946b516f82660118e263f7d2ca9157d20a719ac818"} err="failed to get container status \"2435fdd00d38796a8e2fd6946b516f82660118e263f7d2ca9157d20a719ac818\": rpc error: code = NotFound desc = could not find container \"2435fdd00d38796a8e2fd6946b516f82660118e263f7d2ca9157d20a719ac818\": container with ID starting with 2435fdd00d38796a8e2fd6946b516f82660118e263f7d2ca9157d20a719ac818 not found: ID does not exist"
Sep 16 12:14:50.287981 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.287960    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg6fn\" (UniqueName: \"kubernetes.io/projected/b57b01bf-08fc-4f84-b58e-2a8c34b07cff-kube-api-access-pg6fn\") pod \"b57b01bf-08fc-4f84-b58e-2a8c34b07cff\" (UID: \"b57b01bf-08fc-4f84-b58e-2a8c34b07cff\") "
Sep 16 12:14:50.289994 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.289974    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57b01bf-08fc-4f84-b58e-2a8c34b07cff-kube-api-access-pg6fn" (OuterVolumeSpecName: "kube-api-access-pg6fn") pod "b57b01bf-08fc-4f84-b58e-2a8c34b07cff" (UID: "b57b01bf-08fc-4f84-b58e-2a8c34b07cff"). InnerVolumeSpecName "kube-api-access-pg6fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:14:50.305107 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.305087    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"
Sep 16 12:14:50.381447 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.381423    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sg26x"
Sep 16 12:14:50.388666 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.388643    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg6fn\" (UniqueName: \"kubernetes.io/projected/b57b01bf-08fc-4f84-b58e-2a8c34b07cff-kube-api-access-pg6fn\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:14:50.578088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.578057    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"]
Sep 16 12:14:50.578225 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:14:50.578196    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-tjmf8\" is forbidden: User \"system:node:ip-10-0-136-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-98.ec2.internal' and this object" podUID="b57b01bf-08fc-4f84-b58e-2a8c34b07cff" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:50.581411 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:14:50.581389    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a75d5e_6436_4b60_a85c_0ebd14ae5692.slice/crio-a4721b43333e0f2eb315a61960b945fe24e073ff47a38bd3ec04efa50b15994f WatchSource:0}: Error finding container a4721b43333e0f2eb315a61960b945fe24e073ff47a38bd3ec04efa50b15994f: Status 404 returned error can't find the container with id a4721b43333e0f2eb315a61960b945fe24e073ff47a38bd3ec04efa50b15994f
Sep 16 12:14:50.652040 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:50.652017    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-sg26x"]
Sep 16 12:14:50.654152 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:14:50.654129    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b467b85_e56c_47dc_a9c5_42b2fed0f04d.slice/crio-9d4567e976431f72ecf42b0097821e169e08bf6e3ab646a2f6a3541d7b2e647f WatchSource:0}: Error finding container 9d4567e976431f72ecf42b0097821e169e08bf6e3ab646a2f6a3541d7b2e647f: Status 404 returned error can't find the container with id 9d4567e976431f72ecf42b0097821e169e08bf6e3ab646a2f6a3541d7b2e647f
Sep 16 12:14:51.277060 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:51.276900    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sg26x" event={"ID":"3b467b85-e56c-47dc-a9c5-42b2fed0f04d","Type":"ContainerStarted","Data":"a6976d70dcbb2fe33c30aa961e2378816da9c3498dd0a63a2df3b3871201cb86"}
Sep 16 12:14:51.277060 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:51.276965    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sg26x" event={"ID":"3b467b85-e56c-47dc-a9c5-42b2fed0f04d","Type":"ContainerStarted","Data":"9d4567e976431f72ecf42b0097821e169e08bf6e3ab646a2f6a3541d7b2e647f"}
Sep 16 12:14:51.277060 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:51.277013    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sg26x"
Sep 16 12:14:51.278789 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:51.278766    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s" event={"ID":"b4a75d5e-6436-4b60-a85c-0ebd14ae5692","Type":"ContainerStarted","Data":"a4721b43333e0f2eb315a61960b945fe24e073ff47a38bd3ec04efa50b15994f"}
Sep 16 12:14:51.298907 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:14:51.298861    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-tjmf8\" is forbidden: User \"system:node:ip-10-0-136-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-98.ec2.internal' and this object" podUID="b57b01bf-08fc-4f84-b58e-2a8c34b07cff" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tjmf8"
Sep 16 12:14:51.299306 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:51.299263    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sg26x" podStartSLOduration=1.2992455729999999 podStartE2EDuration="1.299245573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:51.29728297 +0000 UTC m=+688.051241703" watchObservedRunningTime="2026-09-16 12:14:51.299245573 +0000 UTC m=+688.053204306"
Sep 16 12:14:51.760205 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:51.760156    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b57b01bf-08fc-4f84-b58e-2a8c34b07cff" path="/var/lib/kubelet/pods/b57b01bf-08fc-4f84-b58e-2a8c34b07cff/volumes"
Sep 16 12:14:54.299542 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:54.299501    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg" podUID="12a3dc10-068e-4d80-aa2d-3735e3bc3a01" containerName="manager" containerID="cri-o://f564bedd74979fff7ba172cd8f73bd6d6d43a8ea099fbf0e6a9f8e23291f4f61" gracePeriod=2
Sep 16 12:14:54.301340 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:54.301312    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s" event={"ID":"b4a75d5e-6436-4b60-a85c-0ebd14ae5692","Type":"ContainerStarted","Data":"0b4b55fabba841a37f17ea7768a1d8655c26558c41e348fd7bd66e39a64569f7"}
Sep 16 12:14:54.301475 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:54.301384    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"
Sep 16 12:14:54.301475 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:14:54.301377    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-kswdg\" is forbidden: User \"system:node:ip-10-0-136-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-98.ec2.internal' and this object" podUID="12a3dc10-068e-4d80-aa2d-3735e3bc3a01" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:54.302904 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:14:54.302882    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-kswdg\" is forbidden: User \"system:node:ip-10-0-136-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-98.ec2.internal' and this object" podUID="12a3dc10-068e-4d80-aa2d-3735e3bc3a01" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:54.320085 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:54.320043    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s" podStartSLOduration=1.714276457 podStartE2EDuration="5.320029708s" totalImagesPullingTime="3.605753251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:54.317965285 +0000 UTC m=+691.071924018" watchObservedRunningTime="2026-09-16 12:14:54.320029708 +0000 UTC m=+691.073988439"
Sep 16 12:14:54.511556 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:54.511516    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:54.514445 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:14:54.514417    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-kswdg\" is forbidden: User \"system:node:ip-10-0-136-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-98.ec2.internal' and this object" podUID="12a3dc10-068e-4d80-aa2d-3735e3bc3a01" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:54.624087 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:54.623988    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd9dx\" (UniqueName: \"kubernetes.io/projected/12a3dc10-068e-4d80-aa2d-3735e3bc3a01-kube-api-access-zd9dx\") pod \"12a3dc10-068e-4d80-aa2d-3735e3bc3a01\" (UID: \"12a3dc10-068e-4d80-aa2d-3735e3bc3a01\") "
Sep 16 12:14:54.624263 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:54.624159    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12a3dc10-068e-4d80-aa2d-3735e3bc3a01-extensions-socket-volume\") pod \"12a3dc10-068e-4d80-aa2d-3735e3bc3a01\" (UID: \"12a3dc10-068e-4d80-aa2d-3735e3bc3a01\") "
Sep 16 12:14:54.624413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:54.624391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a3dc10-068e-4d80-aa2d-3735e3bc3a01-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "12a3dc10-068e-4d80-aa2d-3735e3bc3a01" (UID: "12a3dc10-068e-4d80-aa2d-3735e3bc3a01"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:14:54.625883 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:54.625864    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a3dc10-068e-4d80-aa2d-3735e3bc3a01-kube-api-access-zd9dx" (OuterVolumeSpecName: "kube-api-access-zd9dx") pod "12a3dc10-068e-4d80-aa2d-3735e3bc3a01" (UID: "12a3dc10-068e-4d80-aa2d-3735e3bc3a01"). InnerVolumeSpecName "kube-api-access-zd9dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:14:54.725739 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:54.725696    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12a3dc10-068e-4d80-aa2d-3735e3bc3a01-extensions-socket-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:14:54.725739 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:54.725733    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd9dx\" (UniqueName: \"kubernetes.io/projected/12a3dc10-068e-4d80-aa2d-3735e3bc3a01-kube-api-access-zd9dx\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:14:55.307305 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:55.307273    2558 generic.go:356] "Generic (PLEG): container finished" podID="12a3dc10-068e-4d80-aa2d-3735e3bc3a01" containerID="f564bedd74979fff7ba172cd8f73bd6d6d43a8ea099fbf0e6a9f8e23291f4f61" exitCode=2
Sep 16 12:14:55.307735 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:55.307309    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:55.307735 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:55.307327    2558 scope.go:117] "RemoveContainer" containerID="f564bedd74979fff7ba172cd8f73bd6d6d43a8ea099fbf0e6a9f8e23291f4f61"
Sep 16 12:14:55.309457 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:14:55.309432    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-kswdg\" is forbidden: User \"system:node:ip-10-0-136-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-98.ec2.internal' and this object" podUID="12a3dc10-068e-4d80-aa2d-3735e3bc3a01" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:55.315596 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:55.315579    2558 scope.go:117] "RemoveContainer" containerID="f564bedd74979fff7ba172cd8f73bd6d6d43a8ea099fbf0e6a9f8e23291f4f61"
Sep 16 12:14:55.315862 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:14:55.315844    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f564bedd74979fff7ba172cd8f73bd6d6d43a8ea099fbf0e6a9f8e23291f4f61\": container with ID starting with f564bedd74979fff7ba172cd8f73bd6d6d43a8ea099fbf0e6a9f8e23291f4f61 not found: ID does not exist" containerID="f564bedd74979fff7ba172cd8f73bd6d6d43a8ea099fbf0e6a9f8e23291f4f61"
Sep 16 12:14:55.315938 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:55.315871    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f564bedd74979fff7ba172cd8f73bd6d6d43a8ea099fbf0e6a9f8e23291f4f61"} err="failed to get container status \"f564bedd74979fff7ba172cd8f73bd6d6d43a8ea099fbf0e6a9f8e23291f4f61\": rpc error: code = NotFound desc = could not find container \"f564bedd74979fff7ba172cd8f73bd6d6d43a8ea099fbf0e6a9f8e23291f4f61\": container with ID starting with f564bedd74979fff7ba172cd8f73bd6d6d43a8ea099fbf0e6a9f8e23291f4f61 not found: ID does not exist"
Sep 16 12:14:55.316930 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:14:55.316908    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-kswdg\" is forbidden: User \"system:node:ip-10-0-136-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-98.ec2.internal' and this object" podUID="12a3dc10-068e-4d80-aa2d-3735e3bc3a01" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kswdg"
Sep 16 12:14:55.757741 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:14:55.757712    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12a3dc10-068e-4d80-aa2d-3735e3bc3a01" path="/var/lib/kubelet/pods/12a3dc10-068e-4d80-aa2d-3735e3bc3a01/volumes"
Sep 16 12:15:02.287625 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:02.287591    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sg26x"
Sep 16 12:15:05.309816 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:05.309780    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"
Sep 16 12:15:18.030532 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.030441    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"]
Sep 16 12:15:18.030992 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.030730    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s" podUID="b4a75d5e-6436-4b60-a85c-0ebd14ae5692" containerName="manager" containerID="cri-o://0b4b55fabba841a37f17ea7768a1d8655c26558c41e348fd7bd66e39a64569f7" gracePeriod=10
Sep 16 12:15:18.272658 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.272633    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"
Sep 16 12:15:18.411661 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.411629    2558 generic.go:356] "Generic (PLEG): container finished" podID="b4a75d5e-6436-4b60-a85c-0ebd14ae5692" containerID="0b4b55fabba841a37f17ea7768a1d8655c26558c41e348fd7bd66e39a64569f7" exitCode=0
Sep 16 12:15:18.411816 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.411670    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"
Sep 16 12:15:18.411816 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.411677    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s" event={"ID":"b4a75d5e-6436-4b60-a85c-0ebd14ae5692","Type":"ContainerDied","Data":"0b4b55fabba841a37f17ea7768a1d8655c26558c41e348fd7bd66e39a64569f7"}
Sep 16 12:15:18.411816 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.411719    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s" event={"ID":"b4a75d5e-6436-4b60-a85c-0ebd14ae5692","Type":"ContainerDied","Data":"a4721b43333e0f2eb315a61960b945fe24e073ff47a38bd3ec04efa50b15994f"}
Sep 16 12:15:18.411816 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.411739    2558 scope.go:117] "RemoveContainer" containerID="0b4b55fabba841a37f17ea7768a1d8655c26558c41e348fd7bd66e39a64569f7"
Sep 16 12:15:18.419539 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.419520    2558 scope.go:117] "RemoveContainer" containerID="0b4b55fabba841a37f17ea7768a1d8655c26558c41e348fd7bd66e39a64569f7"
Sep 16 12:15:18.419807 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:15:18.419788    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4b55fabba841a37f17ea7768a1d8655c26558c41e348fd7bd66e39a64569f7\": container with ID starting with 0b4b55fabba841a37f17ea7768a1d8655c26558c41e348fd7bd66e39a64569f7 not found: ID does not exist" containerID="0b4b55fabba841a37f17ea7768a1d8655c26558c41e348fd7bd66e39a64569f7"
Sep 16 12:15:18.419869 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.419816    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4b55fabba841a37f17ea7768a1d8655c26558c41e348fd7bd66e39a64569f7"} err="failed to get container status \"0b4b55fabba841a37f17ea7768a1d8655c26558c41e348fd7bd66e39a64569f7\": rpc error: code = NotFound desc = could not find container \"0b4b55fabba841a37f17ea7768a1d8655c26558c41e348fd7bd66e39a64569f7\": container with ID starting with 0b4b55fabba841a37f17ea7768a1d8655c26558c41e348fd7bd66e39a64569f7 not found: ID does not exist"
Sep 16 12:15:18.425612 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.425594    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfbkq\" (UniqueName: \"kubernetes.io/projected/b4a75d5e-6436-4b60-a85c-0ebd14ae5692-kube-api-access-sfbkq\") pod \"b4a75d5e-6436-4b60-a85c-0ebd14ae5692\" (UID: \"b4a75d5e-6436-4b60-a85c-0ebd14ae5692\") "
Sep 16 12:15:18.425690 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.425661    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a75d5e-6436-4b60-a85c-0ebd14ae5692-extensions-socket-volume\") pod \"b4a75d5e-6436-4b60-a85c-0ebd14ae5692\" (UID: \"b4a75d5e-6436-4b60-a85c-0ebd14ae5692\") "
Sep 16 12:15:18.426072 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.426051    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a75d5e-6436-4b60-a85c-0ebd14ae5692-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b4a75d5e-6436-4b60-a85c-0ebd14ae5692" (UID: "b4a75d5e-6436-4b60-a85c-0ebd14ae5692"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:15:18.427467 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.427443    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a75d5e-6436-4b60-a85c-0ebd14ae5692-kube-api-access-sfbkq" (OuterVolumeSpecName: "kube-api-access-sfbkq") pod "b4a75d5e-6436-4b60-a85c-0ebd14ae5692" (UID: "b4a75d5e-6436-4b60-a85c-0ebd14ae5692"). InnerVolumeSpecName "kube-api-access-sfbkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:18.526621 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.526589    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a75d5e-6436-4b60-a85c-0ebd14ae5692-extensions-socket-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:15:18.526621 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.526612    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfbkq\" (UniqueName: \"kubernetes.io/projected/b4a75d5e-6436-4b60-a85c-0ebd14ae5692-kube-api-access-sfbkq\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:15:18.739167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.739137    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"]
Sep 16 12:15:18.744851 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.744829    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxd4s"]
Sep 16 12:15:18.766906 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:18.766882    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:19.759217 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:19.759180    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4a75d5e-6436-4b60-a85c-0ebd14ae5692" path="/var/lib/kubelet/pods/b4a75d5e-6436-4b60-a85c-0ebd14ae5692/volumes"
Sep 16 12:15:21.687153 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:21.687121    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:21.782414 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:21.782384    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:26.262331 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:26.262297    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:35.297669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.297632    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ggmvb"]
Sep 16 12:15:35.298137 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.297934    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a3dc10-068e-4d80-aa2d-3735e3bc3a01" containerName="manager"
Sep 16 12:15:35.298137 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.297945    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a3dc10-068e-4d80-aa2d-3735e3bc3a01" containerName="manager"
Sep 16 12:15:35.298137 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.297954    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a75d5e-6436-4b60-a85c-0ebd14ae5692" containerName="manager"
Sep 16 12:15:35.298137 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.297959    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a75d5e-6436-4b60-a85c-0ebd14ae5692" containerName="manager"
Sep 16 12:15:35.298137 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.298013    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a3dc10-068e-4d80-aa2d-3735e3bc3a01" containerName="manager"
Sep 16 12:15:35.298137 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.298022    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a75d5e-6436-4b60-a85c-0ebd14ae5692" containerName="manager"
Sep 16 12:15:35.302130 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.302099    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-ggmvb"
Sep 16 12:15:35.302989 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.302960    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:35.304174 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.304152    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g5zbz\""
Sep 16 12:15:35.304277 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.304153    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 12:15:35.310123 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.310102    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ggmvb"]
Sep 16 12:15:35.354873 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.354840    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/03ecc858-cd41-4e6b-9a91-b90dd767d61d-config-file\") pod \"limitador-limitador-8588749977-ggmvb\" (UID: \"03ecc858-cd41-4e6b-9a91-b90dd767d61d\") " pod="kuadrant-system/limitador-limitador-8588749977-ggmvb"
Sep 16 12:15:35.355042 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.354890    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6nbk\" (UniqueName: \"kubernetes.io/projected/03ecc858-cd41-4e6b-9a91-b90dd767d61d-kube-api-access-k6nbk\") pod \"limitador-limitador-8588749977-ggmvb\" (UID: \"03ecc858-cd41-4e6b-9a91-b90dd767d61d\") " pod="kuadrant-system/limitador-limitador-8588749977-ggmvb"
Sep 16 12:15:35.392159 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.392125    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ggmvb"]
Sep 16 12:15:35.401566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.401475    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:35.455590 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.455528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/03ecc858-cd41-4e6b-9a91-b90dd767d61d-config-file\") pod \"limitador-limitador-8588749977-ggmvb\" (UID: \"03ecc858-cd41-4e6b-9a91-b90dd767d61d\") " pod="kuadrant-system/limitador-limitador-8588749977-ggmvb"
Sep 16 12:15:35.455773 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.455605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6nbk\" (UniqueName: \"kubernetes.io/projected/03ecc858-cd41-4e6b-9a91-b90dd767d61d-kube-api-access-k6nbk\") pod \"limitador-limitador-8588749977-ggmvb\" (UID: \"03ecc858-cd41-4e6b-9a91-b90dd767d61d\") " pod="kuadrant-system/limitador-limitador-8588749977-ggmvb"
Sep 16 12:15:35.456258 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.456239    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/03ecc858-cd41-4e6b-9a91-b90dd767d61d-config-file\") pod \"limitador-limitador-8588749977-ggmvb\" (UID: \"03ecc858-cd41-4e6b-9a91-b90dd767d61d\") " pod="kuadrant-system/limitador-limitador-8588749977-ggmvb"
Sep 16 12:15:35.463869 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.463846    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6nbk\" (UniqueName: \"kubernetes.io/projected/03ecc858-cd41-4e6b-9a91-b90dd767d61d-kube-api-access-k6nbk\") pod \"limitador-limitador-8588749977-ggmvb\" (UID: \"03ecc858-cd41-4e6b-9a91-b90dd767d61d\") " pod="kuadrant-system/limitador-limitador-8588749977-ggmvb"
Sep 16 12:15:35.501426 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.501395    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:35.613025 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.612986    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-ggmvb"
Sep 16 12:15:35.883073 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:35.882989    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ggmvb"]
Sep 16 12:15:35.885633 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:15:35.885600    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ecc858_cd41_4e6b_9a91_b90dd767d61d.slice/crio-29700f99e7299c6c560d485905e2976fe824b6db102969c09f699555442934d7 WatchSource:0}: Error finding container 29700f99e7299c6c560d485905e2976fe824b6db102969c09f699555442934d7: Status 404 returned error can't find the container with id 29700f99e7299c6c560d485905e2976fe824b6db102969c09f699555442934d7
Sep 16 12:15:36.127501 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:36.127470    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:36.190523 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:36.190446    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:36.502929 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:36.502845    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-ggmvb" event={"ID":"03ecc858-cd41-4e6b-9a91-b90dd767d61d","Type":"ContainerStarted","Data":"29700f99e7299c6c560d485905e2976fe824b6db102969c09f699555442934d7"}
Sep 16 12:15:37.508837 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:37.508797    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-ggmvb" event={"ID":"03ecc858-cd41-4e6b-9a91-b90dd767d61d","Type":"ContainerStarted","Data":"08baa2ceffc0f1c1050436cad8d35235ba80e39172748251aa93a4658ac404fd"}
Sep 16 12:15:37.509223 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:37.508873    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-ggmvb"
Sep 16 12:15:37.524386 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:37.524331    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-ggmvb" podStartSLOduration=1.34484133 podStartE2EDuration="2.52431686s" totalImagesPullingTime="1.17947553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:37.522854027 +0000 UTC m=+734.276812755" watchObservedRunningTime="2026-09-16 12:15:37.52431686 +0000 UTC m=+734.278275591"
Sep 16 12:15:48.514605 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:48.514565    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-ggmvb"
Sep 16 12:15:50.053455 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:50.053418    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ggmvb"]
Sep 16 12:15:50.053847 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:50.053606    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-ggmvb" podUID="03ecc858-cd41-4e6b-9a91-b90dd767d61d" containerName="limitador" containerID="cri-o://08baa2ceffc0f1c1050436cad8d35235ba80e39172748251aa93a4658ac404fd" gracePeriod=30
Sep 16 12:15:50.563876 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:50.563849    2558 generic.go:356] "Generic (PLEG): container finished" podID="03ecc858-cd41-4e6b-9a91-b90dd767d61d" containerID="08baa2ceffc0f1c1050436cad8d35235ba80e39172748251aa93a4658ac404fd" exitCode=0
Sep 16 12:15:50.564020 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:50.563921    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-ggmvb" event={"ID":"03ecc858-cd41-4e6b-9a91-b90dd767d61d","Type":"ContainerDied","Data":"08baa2ceffc0f1c1050436cad8d35235ba80e39172748251aa93a4658ac404fd"}
Sep 16 12:15:50.583539 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:50.583517    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-ggmvb"
Sep 16 12:15:50.674810 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:50.674752    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/03ecc858-cd41-4e6b-9a91-b90dd767d61d-config-file\") pod \"03ecc858-cd41-4e6b-9a91-b90dd767d61d\" (UID: \"03ecc858-cd41-4e6b-9a91-b90dd767d61d\") "
Sep 16 12:15:50.674810 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:50.674783    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6nbk\" (UniqueName: \"kubernetes.io/projected/03ecc858-cd41-4e6b-9a91-b90dd767d61d-kube-api-access-k6nbk\") pod \"03ecc858-cd41-4e6b-9a91-b90dd767d61d\" (UID: \"03ecc858-cd41-4e6b-9a91-b90dd767d61d\") "
Sep 16 12:15:50.675155 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:50.675129    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ecc858-cd41-4e6b-9a91-b90dd767d61d-config-file" (OuterVolumeSpecName: "config-file") pod "03ecc858-cd41-4e6b-9a91-b90dd767d61d" (UID: "03ecc858-cd41-4e6b-9a91-b90dd767d61d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:15:50.676722 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:50.676698    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ecc858-cd41-4e6b-9a91-b90dd767d61d-kube-api-access-k6nbk" (OuterVolumeSpecName: "kube-api-access-k6nbk") pod "03ecc858-cd41-4e6b-9a91-b90dd767d61d" (UID: "03ecc858-cd41-4e6b-9a91-b90dd767d61d"). InnerVolumeSpecName "kube-api-access-k6nbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:50.776130 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:50.776104    2558 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/03ecc858-cd41-4e6b-9a91-b90dd767d61d-config-file\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:15:50.776130 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:50.776127    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6nbk\" (UniqueName: \"kubernetes.io/projected/03ecc858-cd41-4e6b-9a91-b90dd767d61d-kube-api-access-k6nbk\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:15:51.569429 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:51.569397    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-ggmvb"
Sep 16 12:15:51.569429 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:51.569424    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-ggmvb" event={"ID":"03ecc858-cd41-4e6b-9a91-b90dd767d61d","Type":"ContainerDied","Data":"29700f99e7299c6c560d485905e2976fe824b6db102969c09f699555442934d7"}
Sep 16 12:15:51.569937 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:51.569477    2558 scope.go:117] "RemoveContainer" containerID="08baa2ceffc0f1c1050436cad8d35235ba80e39172748251aa93a4658ac404fd"
Sep 16 12:15:51.593024 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:51.592998    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ggmvb"]
Sep 16 12:15:51.595246 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:51.595222    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ggmvb"]
Sep 16 12:15:51.756966 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:51.756932    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03ecc858-cd41-4e6b-9a91-b90dd767d61d" path="/var/lib/kubelet/pods/03ecc858-cd41-4e6b-9a91-b90dd767d61d/volumes"
Sep 16 12:15:55.395308 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.395264    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:55.495670 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.495637    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:55.845041 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.845008    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr"]
Sep 16 12:15:55.845399 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.845383    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ecc858-cd41-4e6b-9a91-b90dd767d61d" containerName="limitador"
Sep 16 12:15:55.845438 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.845401    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ecc858-cd41-4e6b-9a91-b90dd767d61d" containerName="limitador"
Sep 16 12:15:55.845472 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.845456    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="03ecc858-cd41-4e6b-9a91-b90dd767d61d" containerName="limitador"
Sep 16 12:15:55.849642 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.849622    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr"
Sep 16 12:15:55.851774 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.851756    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qkjhj\""
Sep 16 12:15:55.851893 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.851875    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:15:55.852490 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.852469    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:15:55.852601 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.852474    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 16 12:15:55.855493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.855471    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr"]
Sep 16 12:15:55.869236 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.869182    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:55.919432 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.919398    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trfh6\" (UniqueName: \"kubernetes.io/projected/69caa2c2-edeb-494f-82be-241487ed81e3-kube-api-access-trfh6\") pod \"postgres-69fb4594cf-8qnhr\" (UID: \"69caa2c2-edeb-494f-82be-241487ed81e3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr"
Sep 16 12:15:55.919586 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:55.919440    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/69caa2c2-edeb-494f-82be-241487ed81e3-data\") pod \"postgres-69fb4594cf-8qnhr\" (UID: \"69caa2c2-edeb-494f-82be-241487ed81e3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr"
Sep 16 12:15:56.020313 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:56.020271    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/69caa2c2-edeb-494f-82be-241487ed81e3-data\") pod \"postgres-69fb4594cf-8qnhr\" (UID: \"69caa2c2-edeb-494f-82be-241487ed81e3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr"
Sep 16 12:15:56.020457 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:56.020353    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trfh6\" (UniqueName: \"kubernetes.io/projected/69caa2c2-edeb-494f-82be-241487ed81e3-kube-api-access-trfh6\") pod \"postgres-69fb4594cf-8qnhr\" (UID: \"69caa2c2-edeb-494f-82be-241487ed81e3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr"
Sep 16 12:15:56.020696 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:56.020675    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/69caa2c2-edeb-494f-82be-241487ed81e3-data\") pod \"postgres-69fb4594cf-8qnhr\" (UID: \"69caa2c2-edeb-494f-82be-241487ed81e3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr"
Sep 16 12:15:56.028127 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:56.028104    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trfh6\" (UniqueName: \"kubernetes.io/projected/69caa2c2-edeb-494f-82be-241487ed81e3-kube-api-access-trfh6\") pod \"postgres-69fb4594cf-8qnhr\" (UID: \"69caa2c2-edeb-494f-82be-241487ed81e3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr"
Sep 16 12:15:56.160929 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:56.160856    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr"
Sep 16 12:15:56.429329 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:56.429308    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr"]
Sep 16 12:15:56.431606 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:15:56.431580    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69caa2c2_edeb_494f_82be_241487ed81e3.slice/crio-d98daeb1dbadeb7618541abab3042616575bff65f9e20f2362fe8cae818f29ff WatchSource:0}: Error finding container d98daeb1dbadeb7618541abab3042616575bff65f9e20f2362fe8cae818f29ff: Status 404 returned error can't find the container with id d98daeb1dbadeb7618541abab3042616575bff65f9e20f2362fe8cae818f29ff
Sep 16 12:15:56.592536 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:15:56.592500    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr" event={"ID":"69caa2c2-edeb-494f-82be-241487ed81e3","Type":"ContainerStarted","Data":"d98daeb1dbadeb7618541abab3042616575bff65f9e20f2362fe8cae818f29ff"}
Sep 16 12:16:01.624852 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:01.624764    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr" event={"ID":"69caa2c2-edeb-494f-82be-241487ed81e3","Type":"ContainerStarted","Data":"7d75301d840718b26fc0f9a03b265e9685547a4eabe98175e93b2de78825d908"}
Sep 16 12:16:01.624852 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:01.624839    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr"
Sep 16 12:16:01.639500 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:01.639445    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr" podStartSLOduration=1.720842029 podStartE2EDuration="6.639428073s" totalImagesPullingTime="4.918586044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:16:01.638111317 +0000 UTC m=+758.392070060" watchObservedRunningTime="2026-09-16 12:16:01.639428073 +0000 UTC m=+758.393386804"
Sep 16 12:16:07.657317 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:07.657283    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8qnhr"
Sep 16 12:16:07.973237 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:07.973164    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:08.328692 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:08.328622    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:15.016903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.016869    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:15.038396 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.038367    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79479fdff4-kfgwm"]
Sep 16 12:16:15.043607 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.043589    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:15.046067 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.046041    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 12:16:15.046252 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.046239    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mpg4c\""
Sep 16 12:16:15.046813 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.046794    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 12:16:15.046893 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.046805    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 12:16:15.054074 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.054053    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79479fdff4-kfgwm"]
Sep 16 12:16:15.115674 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.115643    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:15.178564 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.178515    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-webhook-cert\") pod \"maas-controller-79479fdff4-kfgwm\" (UID: \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\") " pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:15.178734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.178613    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-metrics-certs\") pod \"maas-controller-79479fdff4-kfgwm\" (UID: \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\") " pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:15.178734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.178650    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sqwd\" (UniqueName: \"kubernetes.io/projected/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-kube-api-access-9sqwd\") pod \"maas-controller-79479fdff4-kfgwm\" (UID: \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\") " pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:15.221838 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.221806    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:15.228290 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.228267    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-774dfb4f47-h2hrq"]
Sep 16 12:16:15.231988 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.231968    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:16:15.242560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.242525    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-774dfb4f47-h2hrq"]
Sep 16 12:16:15.279113 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.279051    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-metrics-certs\") pod \"maas-controller-79479fdff4-kfgwm\" (UID: \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\") " pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:15.279113 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.279084    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sqwd\" (UniqueName: \"kubernetes.io/projected/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-kube-api-access-9sqwd\") pod \"maas-controller-79479fdff4-kfgwm\" (UID: \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\") " pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:15.279235 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.279147    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-webhook-cert\") pod \"maas-controller-79479fdff4-kfgwm\" (UID: \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\") " pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:15.279275 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:16:15.279252    2558 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 16 12:16:15.279316 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.279284    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:15.279316 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:16:15.279300    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-webhook-cert podName:55456d8f-ec68-424f-b5fd-b2b9b2757b5c nodeName:}" failed. No retries permitted until 2026-09-16 12:16:15.779287053 +0000 UTC m=+772.533245762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-webhook-cert") pod "maas-controller-79479fdff4-kfgwm" (UID: "55456d8f-ec68-424f-b5fd-b2b9b2757b5c") : secret "maas-controller-webhook-cert" not found
Sep 16 12:16:15.281506 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.281481    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-metrics-certs\") pod \"maas-controller-79479fdff4-kfgwm\" (UID: \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\") " pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:15.289135 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.289114    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sqwd\" (UniqueName: \"kubernetes.io/projected/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-kube-api-access-9sqwd\") pod \"maas-controller-79479fdff4-kfgwm\" (UID: \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\") " pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:15.379961 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.379931    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-webhook-cert\") pod \"maas-controller-774dfb4f47-h2hrq\" (UID: \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\") " pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:16:15.380107 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.380028    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-metrics-certs\") pod \"maas-controller-774dfb4f47-h2hrq\" (UID: \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\") " pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:16:15.380107 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.380056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bnjj\" (UniqueName: \"kubernetes.io/projected/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-kube-api-access-6bnjj\") pod \"maas-controller-774dfb4f47-h2hrq\" (UID: \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\") " pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:16:15.416302 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.416272    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:15.481209 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.481178    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-metrics-certs\") pod \"maas-controller-774dfb4f47-h2hrq\" (UID: \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\") " pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:16:15.481326 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.481213    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnjj\" (UniqueName: \"kubernetes.io/projected/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-kube-api-access-6bnjj\") pod \"maas-controller-774dfb4f47-h2hrq\" (UID: \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\") " pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:16:15.481382 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.481360    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-webhook-cert\") pod \"maas-controller-774dfb4f47-h2hrq\" (UID: \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\") " pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:16:15.483588 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.483568    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-metrics-certs\") pod \"maas-controller-774dfb4f47-h2hrq\" (UID: \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\") " pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:16:15.483689 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.483599    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-webhook-cert\") pod \"maas-controller-774dfb4f47-h2hrq\" (UID: \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\") " pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:16:15.489421 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.489400    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnjj\" (UniqueName: \"kubernetes.io/projected/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-kube-api-access-6bnjj\") pod \"maas-controller-774dfb4f47-h2hrq\" (UID: \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\") " pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:16:15.542329 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.542268    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:16:15.783503 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.783473    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-webhook-cert\") pod \"maas-controller-79479fdff4-kfgwm\" (UID: \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\") " pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:15.785801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.785783    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-webhook-cert\") pod \"maas-controller-79479fdff4-kfgwm\" (UID: \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\") " pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:15.811009 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.810988    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-774dfb4f47-h2hrq"]
Sep 16 12:16:15.813066 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:16:15.813043    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a8d41b_9a6b_42cb_832e_af64d1604ddd.slice/crio-232d8975312fb741a0f08019f7410e53b5b9c86c39b0f02ce2d9535a7c1620d3 WatchSource:0}: Error finding container 232d8975312fb741a0f08019f7410e53b5b9c86c39b0f02ce2d9535a7c1620d3: Status 404 returned error can't find the container with id 232d8975312fb741a0f08019f7410e53b5b9c86c39b0f02ce2d9535a7c1620d3
Sep 16 12:16:15.954129 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:15.954092    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:16.223791 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:16.223764    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79479fdff4-kfgwm"]
Sep 16 12:16:16.225714 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:16:16.225684    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55456d8f_ec68_424f_b5fd_b2b9b2757b5c.slice/crio-cc98085fa15fda26a4f998c0ee375d514da310f20af818c165304d7138129af8 WatchSource:0}: Error finding container cc98085fa15fda26a4f998c0ee375d514da310f20af818c165304d7138129af8: Status 404 returned error can't find the container with id cc98085fa15fda26a4f998c0ee375d514da310f20af818c165304d7138129af8
Sep 16 12:16:16.690807 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:16.690768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79479fdff4-kfgwm" event={"ID":"55456d8f-ec68-424f-b5fd-b2b9b2757b5c","Type":"ContainerStarted","Data":"cc98085fa15fda26a4f998c0ee375d514da310f20af818c165304d7138129af8"}
Sep 16 12:16:16.692230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:16.692200    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774dfb4f47-h2hrq" event={"ID":"f7a8d41b-9a6b-42cb-832e-af64d1604ddd","Type":"ContainerStarted","Data":"232d8975312fb741a0f08019f7410e53b5b9c86c39b0f02ce2d9535a7c1620d3"}
Sep 16 12:16:18.702671 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:18.702636    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79479fdff4-kfgwm" event={"ID":"55456d8f-ec68-424f-b5fd-b2b9b2757b5c","Type":"ContainerStarted","Data":"903b0624555af085f2bccad5b541c305cc60ced5c2a55193b23c8ab58acaa0ec"}
Sep 16 12:16:18.703024 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:18.702759    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:18.704541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:18.704516    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774dfb4f47-h2hrq" event={"ID":"f7a8d41b-9a6b-42cb-832e-af64d1604ddd","Type":"ContainerStarted","Data":"a6a635d0c1ee6e1d1cc9aff56c7d165483e07e71732f4a36ddd3d2a5ece732e8"}
Sep 16 12:16:18.704658 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:18.704603    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:16:18.719062 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:18.718933    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79479fdff4-kfgwm" podStartSLOduration=1.3205182180000001 podStartE2EDuration="3.718918277s" totalImagesPullingTime="2.398400059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:16:18.717531742 +0000 UTC m=+775.471490474" watchObservedRunningTime="2026-09-16 12:16:18.718918277 +0000 UTC m=+775.472877028"
Sep 16 12:16:18.733194 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:18.733141    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-774dfb4f47-h2hrq" podStartSLOduration=0.912793274 podStartE2EDuration="3.733122208s" totalImagesPullingTime="2.820328934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:16:18.732286785 +0000 UTC m=+775.486245517" watchObservedRunningTime="2026-09-16 12:16:18.733122208 +0000 UTC m=+775.487080940"
Sep 16 12:16:18.814579 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:18.814536    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:18.915297 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:18.915200    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:29.710998 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:29.710967    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:16:29.711367 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:29.711040    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:29.768410 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:29.768375    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79479fdff4-kfgwm"]
Sep 16 12:16:29.768692 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:29.768659    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-79479fdff4-kfgwm" podUID="55456d8f-ec68-424f-b5fd-b2b9b2757b5c" containerName="manager" containerID="cri-o://903b0624555af085f2bccad5b541c305cc60ced5c2a55193b23c8ab58acaa0ec" gracePeriod=30
Sep 16 12:16:30.010972 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.010951    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:30.103095 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.103062    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sqwd\" (UniqueName: \"kubernetes.io/projected/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-kube-api-access-9sqwd\") pod \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\" (UID: \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\") "
Sep 16 12:16:30.103256 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.103126    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-metrics-certs\") pod \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\" (UID: \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\") "
Sep 16 12:16:30.103256 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.103159    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-webhook-cert\") pod \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\" (UID: \"55456d8f-ec68-424f-b5fd-b2b9b2757b5c\") "
Sep 16 12:16:30.105186 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.105142    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "55456d8f-ec68-424f-b5fd-b2b9b2757b5c" (UID: "55456d8f-ec68-424f-b5fd-b2b9b2757b5c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:16:30.105299 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.105222    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-kube-api-access-9sqwd" (OuterVolumeSpecName: "kube-api-access-9sqwd") pod "55456d8f-ec68-424f-b5fd-b2b9b2757b5c" (UID: "55456d8f-ec68-424f-b5fd-b2b9b2757b5c"). InnerVolumeSpecName "kube-api-access-9sqwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:16:30.105299 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.105253    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "55456d8f-ec68-424f-b5fd-b2b9b2757b5c" (UID: "55456d8f-ec68-424f-b5fd-b2b9b2757b5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:16:30.204342 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.204301    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-metrics-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:16:30.204342 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.204336    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-webhook-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:16:30.204342 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.204346    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sqwd\" (UniqueName: \"kubernetes.io/projected/55456d8f-ec68-424f-b5fd-b2b9b2757b5c-kube-api-access-9sqwd\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:16:30.760965 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.760931    2558 generic.go:356] "Generic (PLEG): container finished" podID="55456d8f-ec68-424f-b5fd-b2b9b2757b5c" containerID="903b0624555af085f2bccad5b541c305cc60ced5c2a55193b23c8ab58acaa0ec" exitCode=0
Sep 16 12:16:30.761341 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.760972    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79479fdff4-kfgwm"
Sep 16 12:16:30.761341 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.761012    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79479fdff4-kfgwm" event={"ID":"55456d8f-ec68-424f-b5fd-b2b9b2757b5c","Type":"ContainerDied","Data":"903b0624555af085f2bccad5b541c305cc60ced5c2a55193b23c8ab58acaa0ec"}
Sep 16 12:16:30.761341 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.761058    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79479fdff4-kfgwm" event={"ID":"55456d8f-ec68-424f-b5fd-b2b9b2757b5c","Type":"ContainerDied","Data":"cc98085fa15fda26a4f998c0ee375d514da310f20af818c165304d7138129af8"}
Sep 16 12:16:30.761341 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.761082    2558 scope.go:117] "RemoveContainer" containerID="903b0624555af085f2bccad5b541c305cc60ced5c2a55193b23c8ab58acaa0ec"
Sep 16 12:16:30.770659 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.770425    2558 scope.go:117] "RemoveContainer" containerID="903b0624555af085f2bccad5b541c305cc60ced5c2a55193b23c8ab58acaa0ec"
Sep 16 12:16:30.770745 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:16:30.770727    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903b0624555af085f2bccad5b541c305cc60ced5c2a55193b23c8ab58acaa0ec\": container with ID starting with 903b0624555af085f2bccad5b541c305cc60ced5c2a55193b23c8ab58acaa0ec not found: ID does not exist" containerID="903b0624555af085f2bccad5b541c305cc60ced5c2a55193b23c8ab58acaa0ec"
Sep 16 12:16:30.770791 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.770752    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903b0624555af085f2bccad5b541c305cc60ced5c2a55193b23c8ab58acaa0ec"} err="failed to get container status \"903b0624555af085f2bccad5b541c305cc60ced5c2a55193b23c8ab58acaa0ec\": rpc error: code = NotFound desc = could not find container \"903b0624555af085f2bccad5b541c305cc60ced5c2a55193b23c8ab58acaa0ec\": container with ID starting with 903b0624555af085f2bccad5b541c305cc60ced5c2a55193b23c8ab58acaa0ec not found: ID does not exist"
Sep 16 12:16:30.781023 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.781003    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79479fdff4-kfgwm"]
Sep 16 12:16:30.785292 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:30.785272    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-79479fdff4-kfgwm"]
Sep 16 12:16:31.757040 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:31.757006    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55456d8f-ec68-424f-b5fd-b2b9b2757b5c" path="/var/lib/kubelet/pods/55456d8f-ec68-424f-b5fd-b2b9b2757b5c/volumes"
Sep 16 12:16:50.339649 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:50.339561    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:50.428940 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:50.428910    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:51.702058 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.702020    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:51.795904 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.795871    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:51.799319 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.799292    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"]
Sep 16 12:16:51.799639 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.799626    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55456d8f-ec68-424f-b5fd-b2b9b2757b5c" containerName="manager"
Sep 16 12:16:51.799639 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.799639    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="55456d8f-ec68-424f-b5fd-b2b9b2757b5c" containerName="manager"
Sep 16 12:16:51.799730 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.799692    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="55456d8f-ec68-424f-b5fd-b2b9b2757b5c" containerName="manager"
Sep 16 12:16:51.805402 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.805384    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:51.808066 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.808046    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4gkmq\""
Sep 16 12:16:51.808197 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.808112    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:16:51.808197 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.808140    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 12:16:51.814080 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.814053    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"]
Sep 16 12:16:51.880701 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.880667    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhw9t\" (UniqueName: \"kubernetes.io/projected/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-kube-api-access-hhw9t\") pod \"maas-api-75d578b874-9sjxv\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:51.880888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.880710    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-maas-api-tls\") pod \"maas-api-75d578b874-9sjxv\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:51.880888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.880748    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-openshift-service-ca\") pod \"maas-api-75d578b874-9sjxv\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:51.880888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.880817    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-metrics-certs\") pod \"maas-api-75d578b874-9sjxv\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:51.966918 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.966828    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:51.981560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.981523    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-openshift-service-ca\") pod \"maas-api-75d578b874-9sjxv\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:51.981633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.981589    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-metrics-certs\") pod \"maas-api-75d578b874-9sjxv\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:51.981633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.981628    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhw9t\" (UniqueName: \"kubernetes.io/projected/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-kube-api-access-hhw9t\") pod \"maas-api-75d578b874-9sjxv\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:51.981710 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.981662    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-maas-api-tls\") pod \"maas-api-75d578b874-9sjxv\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:51.982148 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.982129    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-openshift-service-ca\") pod \"maas-api-75d578b874-9sjxv\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:51.984231 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.984201    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-metrics-certs\") pod \"maas-api-75d578b874-9sjxv\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:51.984335 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.984252    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-maas-api-tls\") pod \"maas-api-75d578b874-9sjxv\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:51.989204 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:51.989180    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhw9t\" (UniqueName: \"kubernetes.io/projected/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-kube-api-access-hhw9t\") pod \"maas-api-75d578b874-9sjxv\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:52.116785 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.116750    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:52.207644 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.207579    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:52.313477 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.313402    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj"]
Sep 16 12:16:52.313842 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.313781    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:52.316781 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.316765    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj"
Sep 16 12:16:52.318781 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.318761    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 12:16:52.318879 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.318789    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-655rc\""
Sep 16 12:16:52.328375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.328350    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj"]
Sep 16 12:16:52.385045 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.385014    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed56fd2-a5fc-4aa7-a5f3-097d1a308b45-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6lkxj\" (UID: \"eed56fd2-a5fc-4aa7-a5f3-097d1a308b45\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj"
Sep 16 12:16:52.385045 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.385049    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsgs2\" (UniqueName: \"kubernetes.io/projected/eed56fd2-a5fc-4aa7-a5f3-097d1a308b45-kube-api-access-rsgs2\") pod \"payload-pre-processing-59fbc47bf8-6lkxj\" (UID: \"eed56fd2-a5fc-4aa7-a5f3-097d1a308b45\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj"
Sep 16 12:16:52.388845 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.388824    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"]
Sep 16 12:16:52.391520 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:16:52.391492    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a11a9d_b74b_4ae6_b35f_c6bea61d5403.slice/crio-7a082758fff92a94f91cfe6bed75f9839281f1bdcdecf4304471b248013d05d8 WatchSource:0}: Error finding container 7a082758fff92a94f91cfe6bed75f9839281f1bdcdecf4304471b248013d05d8: Status 404 returned error can't find the container with id 7a082758fff92a94f91cfe6bed75f9839281f1bdcdecf4304471b248013d05d8
Sep 16 12:16:52.486196 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.486160    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed56fd2-a5fc-4aa7-a5f3-097d1a308b45-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6lkxj\" (UID: \"eed56fd2-a5fc-4aa7-a5f3-097d1a308b45\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj"
Sep 16 12:16:52.486196 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.486198    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsgs2\" (UniqueName: \"kubernetes.io/projected/eed56fd2-a5fc-4aa7-a5f3-097d1a308b45-kube-api-access-rsgs2\") pod \"payload-pre-processing-59fbc47bf8-6lkxj\" (UID: \"eed56fd2-a5fc-4aa7-a5f3-097d1a308b45\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj"
Sep 16 12:16:52.486864 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.486844    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed56fd2-a5fc-4aa7-a5f3-097d1a308b45-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6lkxj\" (UID: \"eed56fd2-a5fc-4aa7-a5f3-097d1a308b45\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj"
Sep 16 12:16:52.493210 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.493184    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsgs2\" (UniqueName: \"kubernetes.io/projected/eed56fd2-a5fc-4aa7-a5f3-097d1a308b45-kube-api-access-rsgs2\") pod \"payload-pre-processing-59fbc47bf8-6lkxj\" (UID: \"eed56fd2-a5fc-4aa7-a5f3-097d1a308b45\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj"
Sep 16 12:16:52.628274 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.628236    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj"
Sep 16 12:16:52.860718 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.860677    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv" event={"ID":"94a11a9d-b74b-4ae6-b35f-c6bea61d5403","Type":"ContainerStarted","Data":"7a082758fff92a94f91cfe6bed75f9839281f1bdcdecf4304471b248013d05d8"}
Sep 16 12:16:52.897917 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:52.897770    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj"]
Sep 16 12:16:52.900447 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:16:52.900420    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed56fd2_a5fc_4aa7_a5f3_097d1a308b45.slice/crio-dc8c566131e8d291736cb3c617204ac97159f19c52ece9dbfcd87040a693b950 WatchSource:0}: Error finding container dc8c566131e8d291736cb3c617204ac97159f19c52ece9dbfcd87040a693b950: Status 404 returned error can't find the container with id dc8c566131e8d291736cb3c617204ac97159f19c52ece9dbfcd87040a693b950
Sep 16 12:16:53.867787 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:53.867743    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj" event={"ID":"eed56fd2-a5fc-4aa7-a5f3-097d1a308b45","Type":"ContainerStarted","Data":"dc8c566131e8d291736cb3c617204ac97159f19c52ece9dbfcd87040a693b950"}
Sep 16 12:16:54.873887 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:54.873854    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj" event={"ID":"eed56fd2-a5fc-4aa7-a5f3-097d1a308b45","Type":"ContainerStarted","Data":"387dbc03c17f8c1a7fed04b7e9c5a1af85cdd137c8c44bd7cb34366e112ab57a"}
Sep 16 12:16:54.874357 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:54.873961    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj"
Sep 16 12:16:54.875419 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:54.875391    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv" event={"ID":"94a11a9d-b74b-4ae6-b35f-c6bea61d5403","Type":"ContainerStarted","Data":"442f784ae354c3164a39c20dee3a7173b874085620725da8f60a70d68de386dc"}
Sep 16 12:16:54.875535 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:54.875444    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:16:54.889520 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:54.889471    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj" podStartSLOduration=1.435019074 podStartE2EDuration="2.889454075s" totalImagesPullingTime="1.454435001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:16:54.888988317 +0000 UTC m=+811.642947049" watchObservedRunningTime="2026-09-16 12:16:54.889454075 +0000 UTC m=+811.643412809"
Sep 16 12:16:54.903732 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:16:54.903689    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv" podStartSLOduration=1.9423756650000001 podStartE2EDuration="3.903675961s" totalImagesPullingTime="1.961300296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:16:54.902104363 +0000 UTC m=+811.656063095" watchObservedRunningTime="2026-09-16 12:16:54.903675961 +0000 UTC m=+811.657634692"
Sep 16 12:17:00.885461 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:00.885425    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:17:01.883837 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:01.883797    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:01.978874 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:01.978841    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:03.042304 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:03.042267    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:05.882728 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:05.882694    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkxj"
Sep 16 12:17:06.202769 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:06.202691    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:10.547992 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.547960    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"]
Sep 16 12:17:10.551810 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.551789    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.554074 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.554038    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 12:17:10.554202 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.554077    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 12:17:10.554721 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.554704    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 16 12:17:10.554803 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.554764    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c49mq\""
Sep 16 12:17:10.562189 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.562160    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"]
Sep 16 12:17:10.575784 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.575728    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:10.637969 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.637932    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92df483b-6db3-43ce-a2e5-1f0549ff5346-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.638151 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.638021    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92df483b-6db3-43ce-a2e5-1f0549ff5346-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.638151 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.638050    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxs7\" (UniqueName: \"kubernetes.io/projected/92df483b-6db3-43ce-a2e5-1f0549ff5346-kube-api-access-wqxs7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.638151 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.638100    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92df483b-6db3-43ce-a2e5-1f0549ff5346-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.638151 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.638124    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92df483b-6db3-43ce-a2e5-1f0549ff5346-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.638337 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.638180    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92df483b-6db3-43ce-a2e5-1f0549ff5346-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.738876 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.738841    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92df483b-6db3-43ce-a2e5-1f0549ff5346-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.738876 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.738879    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxs7\" (UniqueName: \"kubernetes.io/projected/92df483b-6db3-43ce-a2e5-1f0549ff5346-kube-api-access-wqxs7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.739089 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.738926    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92df483b-6db3-43ce-a2e5-1f0549ff5346-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.739089 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.738943    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92df483b-6db3-43ce-a2e5-1f0549ff5346-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.739089 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.738980    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92df483b-6db3-43ce-a2e5-1f0549ff5346-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.739089 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.739004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92df483b-6db3-43ce-a2e5-1f0549ff5346-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.739378 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.739354    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92df483b-6db3-43ce-a2e5-1f0549ff5346-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.739417 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.739391    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92df483b-6db3-43ce-a2e5-1f0549ff5346-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.739482 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.739439    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92df483b-6db3-43ce-a2e5-1f0549ff5346-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.741327 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.741308    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92df483b-6db3-43ce-a2e5-1f0549ff5346-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.741414 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.741399    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92df483b-6db3-43ce-a2e5-1f0549ff5346-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.746253 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.746227    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxs7\" (UniqueName: \"kubernetes.io/projected/92df483b-6db3-43ce-a2e5-1f0549ff5346-kube-api-access-wqxs7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb\" (UID: \"92df483b-6db3-43ce-a2e5-1f0549ff5346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:10.863437 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:10.863398    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:11.134929 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:11.134833    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"]
Sep 16 12:17:11.137880 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:17:11.137847    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92df483b_6db3_43ce_a2e5_1f0549ff5346.slice/crio-05e61a61c9fa694fc4ab214cf749a295276dfe6e0ad1e398499f18ee76524985 WatchSource:0}: Error finding container 05e61a61c9fa694fc4ab214cf749a295276dfe6e0ad1e398499f18ee76524985: Status 404 returned error can't find the container with id 05e61a61c9fa694fc4ab214cf749a295276dfe6e0ad1e398499f18ee76524985
Sep 16 12:17:11.961998 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:11.961956    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb" event={"ID":"92df483b-6db3-43ce-a2e5-1f0549ff5346","Type":"ContainerStarted","Data":"05e61a61c9fa694fc4ab214cf749a295276dfe6e0ad1e398499f18ee76524985"}
Sep 16 12:17:12.967814 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:12.967772    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb" event={"ID":"92df483b-6db3-43ce-a2e5-1f0549ff5346","Type":"ContainerStarted","Data":"022dda934dfdd24738ba980ed8ba3ef7ca3e752599db15bd77dd1f85538a0712"}
Sep 16 12:17:12.984197 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:12.984140    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb" podStartSLOduration=2.01151594 podStartE2EDuration="2.984123623s" totalImagesPullingTime="972.607683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:17:12.982874289 +0000 UTC m=+829.736833020" watchObservedRunningTime="2026-09-16 12:17:12.984123623 +0000 UTC m=+829.738082385"
Sep 16 12:17:20.864163 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:20.864110    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:20.864716 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:20.864190    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:20.876791 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:20.876761    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:21.017781 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:21.017746    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb"
Sep 16 12:17:35.370905 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:35.370870    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:47.541150 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:47.541111    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:58.749396 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:17:58.749354    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:08.890725 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:08.890646    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:17.231654 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.231578    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"]
Sep 16 12:18:17.235916 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.235892    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.238005 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.237985    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 16 12:18:17.243354 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.243330    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"]
Sep 16 12:18:17.274490 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.274460    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:17.413730 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.413693    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e491c07-b1ad-4746-862f-dae40435fd76-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.413921 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.413741    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrr2x\" (UniqueName: \"kubernetes.io/projected/8e491c07-b1ad-4746-862f-dae40435fd76-kube-api-access-xrr2x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.413921 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.413761    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e491c07-b1ad-4746-862f-dae40435fd76-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.413921 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.413858    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e491c07-b1ad-4746-862f-dae40435fd76-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.413921 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.413902    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e491c07-b1ad-4746-862f-dae40435fd76-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.414062 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.413954    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e491c07-b1ad-4746-862f-dae40435fd76-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.514765 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.514661    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e491c07-b1ad-4746-862f-dae40435fd76-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.514765 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.514732    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e491c07-b1ad-4746-862f-dae40435fd76-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.514997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.514774    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrr2x\" (UniqueName: \"kubernetes.io/projected/8e491c07-b1ad-4746-862f-dae40435fd76-kube-api-access-xrr2x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.514997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.514802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e491c07-b1ad-4746-862f-dae40435fd76-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.514997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.514844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e491c07-b1ad-4746-862f-dae40435fd76-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.514997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.514869    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e491c07-b1ad-4746-862f-dae40435fd76-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.515186 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.515154    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e491c07-b1ad-4746-862f-dae40435fd76-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.515243 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.515198    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e491c07-b1ad-4746-862f-dae40435fd76-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.515299 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.515239    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e491c07-b1ad-4746-862f-dae40435fd76-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.517078 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.517056    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e491c07-b1ad-4746-862f-dae40435fd76-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.517167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.517131    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e491c07-b1ad-4746-862f-dae40435fd76-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.522242 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.522224    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrr2x\" (UniqueName: \"kubernetes.io/projected/8e491c07-b1ad-4746-862f-dae40435fd76-kube-api-access-xrr2x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tzkw9\" (UID: \"8e491c07-b1ad-4746-862f-dae40435fd76\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.548781 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.548740    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:17.817167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:17.817137    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"]
Sep 16 12:18:17.819580 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:18:17.819531    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e491c07_b1ad_4746_862f_dae40435fd76.slice/crio-00d87048e65d37843723788c3683caf35b293339faffcd520f043762bf19480d WatchSource:0}: Error finding container 00d87048e65d37843723788c3683caf35b293339faffcd520f043762bf19480d: Status 404 returned error can't find the container with id 00d87048e65d37843723788c3683caf35b293339faffcd520f043762bf19480d
Sep 16 12:18:18.267163 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:18.267123    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9" event={"ID":"8e491c07-b1ad-4746-862f-dae40435fd76","Type":"ContainerStarted","Data":"05cf3bca86799f70ddce60bf609fed3babb2a5ac53059ad92eefb08eb3db2997"}
Sep 16 12:18:18.267619 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:18.267181    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9" event={"ID":"8e491c07-b1ad-4746-862f-dae40435fd76","Type":"ContainerStarted","Data":"00d87048e65d37843723788c3683caf35b293339faffcd520f043762bf19480d"}
Sep 16 12:18:18.284885 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:18.284827    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9" podStartSLOduration=1.095730354 podStartE2EDuration="1.284809165s" totalImagesPullingTime="189.078811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:18:18.282571797 +0000 UTC m=+895.036530519" watchObservedRunningTime="2026-09-16 12:18:18.284809165 +0000 UTC m=+895.038767896"
Sep 16 12:18:23.767764 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:23.767724    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:27.549337 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:27.549303    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:27.549701 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:27.549385    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:27.561631 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:27.561610    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:28.326197 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:28.326159    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tzkw9"
Sep 16 12:18:38.404838 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:38.404806    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:38.455611 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:18:38.455585    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:40.842244 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:19:40.842207    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:42.372950 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:19:42.372919    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:00.455594 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.455478    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:00.507685 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.507650    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:00.574661 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.574626    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:00.806822 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.806728    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:00.827958 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.827929    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"]
Sep 16 12:20:00.831314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.831288    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:00.836054 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.836035    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-68f7cd-openshift-default-dockercfg-gnwnb\""
Sep 16 12:20:00.844355 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.844328    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"]
Sep 16 12:20:00.899331 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.899285    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:00.992432 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.992396    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:00.992663 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.992644    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-envoy\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:00.992734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.992680    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-credential-socket\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:00.992734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.992708    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-workload-certs\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:00.992734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.992728    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-workload-socket\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:00.992890 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.992800    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a44004-4f18-4869-9290-faa4f6b4a9cf-istiod-ca-cert\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:00.992890 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.992836    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-podinfo\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:00.992890 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.992865    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmvs2\" (UniqueName: \"kubernetes.io/projected/04a44004-4f18-4869-9290-faa4f6b4a9cf-kube-api-access-lmvs2\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:00.993005 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.992966    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-data\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:00.993005 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:00.992985    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-token\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.092983 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.092945    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:01.094225 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.094197    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-credential-socket\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.094310 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.094229    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-workload-certs\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.094310 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.094248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-workload-socket\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.094425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.094308    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a44004-4f18-4869-9290-faa4f6b4a9cf-istiod-ca-cert\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.094425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.094364    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-podinfo\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.094425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.094414    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmvs2\" (UniqueName: \"kubernetes.io/projected/04a44004-4f18-4869-9290-faa4f6b4a9cf-kube-api-access-lmvs2\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.094613 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.094480    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-data\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.094613 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.094580    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-workload-socket\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.094719 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.094619    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-token\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.094719 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.094707    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-envoy\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.094819 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.094782    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-workload-certs\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.094876 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.094840    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-credential-socket\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.094876 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.094856    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-data\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.095187 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.095107    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a44004-4f18-4869-9290-faa4f6b4a9cf-istiod-ca-cert\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.096966 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.096943    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-envoy\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.097057 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.097023    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-podinfo\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.105444 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.105420    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmvs2\" (UniqueName: \"kubernetes.io/projected/04a44004-4f18-4869-9290-faa4f6b4a9cf-kube-api-access-lmvs2\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.105542 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.105525    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-token\") pod \"e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") " pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.148825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.148788    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:01.193203 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.193171    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:01.406500 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.406411    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:01.424379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.424354    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"]
Sep 16 12:20:01.426774 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:20:01.426748    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a44004_4f18_4869_9290_faa4f6b4a9cf.slice/crio-616a50cac4c2d5619dd6348cc9f3b378dfad944e5a951b1b2a43a0e385ce0b11 WatchSource:0}: Error finding container 616a50cac4c2d5619dd6348cc9f3b378dfad944e5a951b1b2a43a0e385ce0b11: Status 404 returned error can't find the container with id 616a50cac4c2d5619dd6348cc9f3b378dfad944e5a951b1b2a43a0e385ce0b11
Sep 16 12:20:01.428913 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.428893    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:20:01.736811 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.736725    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" event={"ID":"04a44004-4f18-4869-9290-faa4f6b4a9cf","Type":"ContainerStarted","Data":"616a50cac4c2d5619dd6348cc9f3b378dfad944e5a951b1b2a43a0e385ce0b11"}
Sep 16 12:20:01.931413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:01.931372    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:02.032754 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:02.032658    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:02.370376 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:02.370338    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:03.543507 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:03.543463    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:03.649967 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:03.649925    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:04.040941 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:04.040897    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:04.041073 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:04.040994    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:04.041073 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:04.041035    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:04.596332 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:04.596280    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:04.691149 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:04.691113    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:04.753593 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:04.753558    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" event={"ID":"04a44004-4f18-4869-9290-faa4f6b4a9cf","Type":"ContainerStarted","Data":"e59e49b506d9b959ad6e7cdb45e9d562cb4a881719b4b195ea75eec32e8adde2"}
Sep 16 12:20:04.773202 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:04.773140    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" podStartSLOduration=2.161614948 podStartE2EDuration="4.773121739s" totalImagesPullingTime="2.611506791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:04.771302433 +0000 UTC m=+1001.525261164" watchObservedRunningTime="2026-09-16 12:20:04.773121739 +0000 UTC m=+1001.527080467"
Sep 16 12:20:04.914578 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:04.914471    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"]
Sep 16 12:20:04.917963 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:04.917946    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:04.919916 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:04.919890    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f79e22-serving-cert\""
Sep 16 12:20:04.927042 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:04.926987    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"]
Sep 16 12:20:04.950687 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:04.950653    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:05.032334 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.032059    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b1f827-a3be-4282-b03a-2132d3b62edd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:05.032334 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.032109    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rd4\" (UniqueName: \"kubernetes.io/projected/b4b1f827-a3be-4282-b03a-2132d3b62edd-kube-api-access-g6rd4\") pod \"maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:05.032334 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.032145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4b1f827-a3be-4282-b03a-2132d3b62edd-maas-api-tls\") pod \"maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:05.032334 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.032216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4b1f827-a3be-4282-b03a-2132d3b62edd-metrics-certs\") pod \"maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:05.039390 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.039358    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:05.133509 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.133477    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4b1f827-a3be-4282-b03a-2132d3b62edd-metrics-certs\") pod \"maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:05.133704 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.133584    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b1f827-a3be-4282-b03a-2132d3b62edd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:05.133704 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.133617    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rd4\" (UniqueName: \"kubernetes.io/projected/b4b1f827-a3be-4282-b03a-2132d3b62edd-kube-api-access-g6rd4\") pod \"maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:05.133704 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.133646    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4b1f827-a3be-4282-b03a-2132d3b62edd-maas-api-tls\") pod \"maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:05.134324 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.134287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b1f827-a3be-4282-b03a-2132d3b62edd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:05.136177 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.136153    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4b1f827-a3be-4282-b03a-2132d3b62edd-metrics-certs\") pod \"maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:05.136328 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.136306    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4b1f827-a3be-4282-b03a-2132d3b62edd-maas-api-tls\") pod \"maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:05.145779 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.145756    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rd4\" (UniqueName: \"kubernetes.io/projected/b4b1f827-a3be-4282-b03a-2132d3b62edd-kube-api-access-g6rd4\") pod \"maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:05.149323 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.149297    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:05.154695 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.154671    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:05.203211 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.203117    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:05.229861 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.229828    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:05.501109 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.501080    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"]
Sep 16 12:20:05.503679 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:20:05.503653    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b1f827_a3be_4282_b03a_2132d3b62edd.slice/crio-b03a1680438306c3f72a76100cc5dfc47d81e28cb00687a853b30c2f3cdd6f97 WatchSource:0}: Error finding container b03a1680438306c3f72a76100cc5dfc47d81e28cb00687a853b30c2f3cdd6f97: Status 404 returned error can't find the container with id b03a1680438306c3f72a76100cc5dfc47d81e28cb00687a853b30c2f3cdd6f97
Sep 16 12:20:05.650965 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.650933    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:05.727843 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.727801    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"]
Sep 16 12:20:05.731780 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.731746    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:05.732604 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.732584    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"
Sep 16 12:20:05.735452 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.735427    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f79e22-dockercfg-pmplg\""
Sep 16 12:20:05.735697 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.735680    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f79e22\""
Sep 16 12:20:05.745062 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.745040    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"]
Sep 16 12:20:05.759649 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.759563    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq" event={"ID":"b4b1f827-a3be-4282-b03a-2132d3b62edd","Type":"ContainerStarted","Data":"b03a1680438306c3f72a76100cc5dfc47d81e28cb00687a853b30c2f3cdd6f97"}
Sep 16 12:20:05.759922 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.759893    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:05.760843 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.760813    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:05.840084 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.840051    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb135408-2e81-4cca-9522-92d16cddb49d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk\" (UID: \"fb135408-2e81-4cca-9522-92d16cddb49d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"
Sep 16 12:20:05.840250 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.840095    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tntkq\" (UniqueName: \"kubernetes.io/projected/fb135408-2e81-4cca-9522-92d16cddb49d-kube-api-access-tntkq\") pod \"payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk\" (UID: \"fb135408-2e81-4cca-9522-92d16cddb49d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"
Sep 16 12:20:05.941318 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.941274    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb135408-2e81-4cca-9522-92d16cddb49d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk\" (UID: \"fb135408-2e81-4cca-9522-92d16cddb49d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"
Sep 16 12:20:05.941476 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.941344    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntkq\" (UniqueName: \"kubernetes.io/projected/fb135408-2e81-4cca-9522-92d16cddb49d-kube-api-access-tntkq\") pod \"payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk\" (UID: \"fb135408-2e81-4cca-9522-92d16cddb49d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"
Sep 16 12:20:05.942162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.942142    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb135408-2e81-4cca-9522-92d16cddb49d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk\" (UID: \"fb135408-2e81-4cca-9522-92d16cddb49d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"
Sep 16 12:20:05.948992 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:05.948965    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntkq\" (UniqueName: \"kubernetes.io/projected/fb135408-2e81-4cca-9522-92d16cddb49d-kube-api-access-tntkq\") pod \"payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk\" (UID: \"fb135408-2e81-4cca-9522-92d16cddb49d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"
Sep 16 12:20:06.039185 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.039069    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:06.044118 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.044091    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"
Sep 16 12:20:06.318827 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.318801    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"]
Sep 16 12:20:06.320670 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:20:06.320639    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb135408_2e81_4cca_9522_92d16cddb49d.slice/crio-3fafdebdf464b5d37aa40fecfaf0af710e3b21bd20a64b6c02ddd89bfcc425ce WatchSource:0}: Error finding container 3fafdebdf464b5d37aa40fecfaf0af710e3b21bd20a64b6c02ddd89bfcc425ce: Status 404 returned error can't find the container with id 3fafdebdf464b5d37aa40fecfaf0af710e3b21bd20a64b6c02ddd89bfcc425ce
Sep 16 12:20:06.766937 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.766884    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq" event={"ID":"b4b1f827-a3be-4282-b03a-2132d3b62edd","Type":"ContainerStarted","Data":"12d1151f3663e1822834efd4aa60c7de071fd5cd5981a18b026533f82315a8de"}
Sep 16 12:20:06.767433 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.767057    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:06.768943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.768912    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk" event={"ID":"fb135408-2e81-4cca-9522-92d16cddb49d","Type":"ContainerStarted","Data":"892aaad2350ca858d9d8edc4f93586d522d52a10afad729becf57e565e00bcce"}
Sep 16 12:20:06.769054 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.768965    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk" event={"ID":"fb135408-2e81-4cca-9522-92d16cddb49d","Type":"ContainerStarted","Data":"3fafdebdf464b5d37aa40fecfaf0af710e3b21bd20a64b6c02ddd89bfcc425ce"}
Sep 16 12:20:06.786415 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.786366    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq" podStartSLOduration=2.0599612 podStartE2EDuration="2.786352372s" totalImagesPullingTime="726.391172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:06.784628114 +0000 UTC m=+1003.538586848" watchObservedRunningTime="2026-09-16 12:20:06.786352372 +0000 UTC m=+1003.540311103"
Sep 16 12:20:06.803912 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.803861    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk" podStartSLOduration=1.803847159 podStartE2EDuration="1.803847159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:06.801839079 +0000 UTC m=+1003.555797810" watchObservedRunningTime="2026-09-16 12:20:06.803847159 +0000 UTC m=+1003.557805890"
Sep 16 12:20:06.905336 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.905301    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"]
Sep 16 12:20:06.908943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.908915    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:06.911027 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.910995    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-739ee7-serving-cert\""
Sep 16 12:20:06.923342 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.923231    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"]
Sep 16 12:20:06.925674 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:06.925642    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:07.020146 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.020048    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:07.051282 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.051240    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.051456 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.051299    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-metrics-certs\") pod \"maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.051456 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.051383    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hsxp\" (UniqueName: \"kubernetes.io/projected/e3e32a8c-2a69-4693-a45f-709fc95ce47f-kube-api-access-2hsxp\") pod \"maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.051456 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.051449    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e32a8c-2a69-4693-a45f-709fc95ce47f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.152893 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.152850    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hsxp\" (UniqueName: \"kubernetes.io/projected/e3e32a8c-2a69-4693-a45f-709fc95ce47f-kube-api-access-2hsxp\") pod \"maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.153044 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.152962    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e32a8c-2a69-4693-a45f-709fc95ce47f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.153044 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.152990    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.153044 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.153020    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-metrics-certs\") pod \"maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.153249 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:20:07.153134    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-serving-cert: secret "maas-api-e2e-worker-w0-739ee7-serving-cert" not found
Sep 16 12:20:07.153249 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:20:07.153223    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-maas-api-tls podName:e3e32a8c-2a69-4693-a45f-709fc95ce47f nodeName:}" failed. No retries permitted until 2026-09-16 12:20:07.653201104 +0000 UTC m=+1004.407159827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-maas-api-tls") pod "maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx" (UID: "e3e32a8c-2a69-4693-a45f-709fc95ce47f") : secret "maas-api-e2e-worker-w0-739ee7-serving-cert" not found
Sep 16 12:20:07.153786 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.153762    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e32a8c-2a69-4693-a45f-709fc95ce47f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.155423 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.155399    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-metrics-certs\") pod \"maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.160533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.160516    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hsxp\" (UniqueName: \"kubernetes.io/projected/e3e32a8c-2a69-4693-a45f-709fc95ce47f-kube-api-access-2hsxp\") pod \"maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.512057 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.512021    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:07.611906 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.611867    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:07.658009 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.657975    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.660263 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.660242    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.776124 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.776031    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"
Sep 16 12:20:07.827437 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.827401    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:07.938674 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:07.938639    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:08.100147 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:08.100071    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"]
Sep 16 12:20:08.102378 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:20:08.102342    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e32a8c_2a69_4693_a45f_709fc95ce47f.slice/crio-51bfd4e6e89b69d644cfd190ad2ebf69b05d4f5903a2ff3fd6392d45a81a24e3 WatchSource:0}: Error finding container 51bfd4e6e89b69d644cfd190ad2ebf69b05d4f5903a2ff3fd6392d45a81a24e3: Status 404 returned error can't find the container with id 51bfd4e6e89b69d644cfd190ad2ebf69b05d4f5903a2ff3fd6392d45a81a24e3
Sep 16 12:20:08.656956 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:08.656858    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:08.782525 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:08.782486    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx" event={"ID":"e3e32a8c-2a69-4693-a45f-709fc95ce47f","Type":"ContainerStarted","Data":"42ba498b038be7cea9104c91d8c278c8911b3918445aecd0232df510a10047cd"}
Sep 16 12:20:08.783061 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:08.782582    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx" event={"ID":"e3e32a8c-2a69-4693-a45f-709fc95ce47f","Type":"ContainerStarted","Data":"51bfd4e6e89b69d644cfd190ad2ebf69b05d4f5903a2ff3fd6392d45a81a24e3"}
Sep 16 12:20:08.783061 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:08.782803    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:08.808799 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:08.808743    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx" podStartSLOduration=2.534725328 podStartE2EDuration="2.808723847s" totalImagesPullingTime="273.998519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:08.807270887 +0000 UTC m=+1005.561229631" watchObservedRunningTime="2026-09-16 12:20:08.808723847 +0000 UTC m=+1005.562682579"
Sep 16 12:20:08.914848 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:08.914746    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:09.010120 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.010081    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:09.231936 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.231849    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:09.337438 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.337411    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:09.462875 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.462843    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:09.530782 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.530707    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:09.592586 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.592530    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"]
Sep 16 12:20:09.596727 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.596682    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:09.599196 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.599175    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-68f7cd-serving-cert\""
Sep 16 12:20:09.604533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.604507    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"]
Sep 16 12:20:09.626457 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.626413    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:09.731180 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.731144    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:09.778137 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.778099    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bd7bbf-7580-4df8-a82a-3c49f018386b-metrics-certs\") pod \"maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:09.778300 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.778174    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5z8p\" (UniqueName: \"kubernetes.io/projected/04bd7bbf-7580-4df8-a82a-3c49f018386b-kube-api-access-w5z8p\") pod \"maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:09.778300 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.778270    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bd7bbf-7580-4df8-a82a-3c49f018386b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:09.778392 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.778358    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bd7bbf-7580-4df8-a82a-3c49f018386b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:09.795239 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.795121    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:09.879533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.879492    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bd7bbf-7580-4df8-a82a-3c49f018386b-metrics-certs\") pod \"maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:09.879719 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.879585    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5z8p\" (UniqueName: \"kubernetes.io/projected/04bd7bbf-7580-4df8-a82a-3c49f018386b-kube-api-access-w5z8p\") pod \"maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:09.879719 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.879695    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bd7bbf-7580-4df8-a82a-3c49f018386b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:09.879847 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.879833    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bd7bbf-7580-4df8-a82a-3c49f018386b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:09.880441 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.880405    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bd7bbf-7580-4df8-a82a-3c49f018386b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:09.882192 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.882117    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bd7bbf-7580-4df8-a82a-3c49f018386b-metrics-certs\") pod \"maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:09.882192 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.882139    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bd7bbf-7580-4df8-a82a-3c49f018386b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:09.887615 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.887584    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5z8p\" (UniqueName: \"kubernetes.io/projected/04bd7bbf-7580-4df8-a82a-3c49f018386b-kube-api-access-w5z8p\") pod \"maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:09.909267 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:09.909237    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:10.050281 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:10.050200    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:10.145343 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:10.145313    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:10.183535 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:10.183513    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"]
Sep 16 12:20:10.185584 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:20:10.185504    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bd7bbf_7580_4df8_a82a_3c49f018386b.slice/crio-5e417a094f61d0c36e8a8d73c0cd081650dc9f7756aeb0656f6147555771210f WatchSource:0}: Error finding container 5e417a094f61d0c36e8a8d73c0cd081650dc9f7756aeb0656f6147555771210f: Status 404 returned error can't find the container with id 5e417a094f61d0c36e8a8d73c0cd081650dc9f7756aeb0656f6147555771210f
Sep 16 12:20:10.227656 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:10.227611    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:10.794527 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:10.794432    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q" event={"ID":"04bd7bbf-7580-4df8-a82a-3c49f018386b","Type":"ContainerStarted","Data":"5a8437491376fb309c3a8e19264d01008d9a3cf0ffa996195bc49bb6f6f1a32b"}
Sep 16 12:20:10.794527 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:10.794493    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q" event={"ID":"04bd7bbf-7580-4df8-a82a-3c49f018386b","Type":"ContainerStarted","Data":"5e417a094f61d0c36e8a8d73c0cd081650dc9f7756aeb0656f6147555771210f"}
Sep 16 12:20:10.794733 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:10.794614    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:10.829931 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:10.829878    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q" podStartSLOduration=1.4914486 podStartE2EDuration="1.829862049s" totalImagesPullingTime="338.413449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:10.827981395 +0000 UTC m=+1007.581940127" watchObservedRunningTime="2026-09-16 12:20:10.829862049 +0000 UTC m=+1007.583820781"
Sep 16 12:20:11.432377 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:11.432342    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:12.000892 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:12.000856    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:12.100783 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:12.100750    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:12.379127 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:12.379084    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:12.479062 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:12.479016    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:12.579840 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:12.579787    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:12.781128 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:12.781059    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:20:13.507483 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:13.507452    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:14.425951 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:14.425917    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:14.794409 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:14.794328    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:20:15.034296 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.034249    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"]
Sep 16 12:20:15.038835 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.038807    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.041341 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.041316    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-00a141-serving-cert\""
Sep 16 12:20:15.048256 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.048141    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"]
Sep 16 12:20:15.082416 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.082384    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:15.126791 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.126751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244e6664-8f6f-4019-b8b7-bf86f29e474b-openshift-service-ca\") pod \"maas-api-e2e-autores-00a141-9bdf57677-9rxcm\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.126791 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.126793    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-metrics-certs\") pod \"maas-api-e2e-autores-00a141-9bdf57677-9rxcm\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.127001 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.126815    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-maas-api-tls\") pod \"maas-api-e2e-autores-00a141-9bdf57677-9rxcm\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.127001 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.126857    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5h5b\" (UniqueName: \"kubernetes.io/projected/244e6664-8f6f-4019-b8b7-bf86f29e474b-kube-api-access-m5h5b\") pod \"maas-api-e2e-autores-00a141-9bdf57677-9rxcm\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.181238 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.181190    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:15.227620 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.227578    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244e6664-8f6f-4019-b8b7-bf86f29e474b-openshift-service-ca\") pod \"maas-api-e2e-autores-00a141-9bdf57677-9rxcm\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.227748 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.227632    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-metrics-certs\") pod \"maas-api-e2e-autores-00a141-9bdf57677-9rxcm\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.227748 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.227653    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-maas-api-tls\") pod \"maas-api-e2e-autores-00a141-9bdf57677-9rxcm\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.227748 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.227674    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5h5b\" (UniqueName: \"kubernetes.io/projected/244e6664-8f6f-4019-b8b7-bf86f29e474b-kube-api-access-m5h5b\") pod \"maas-api-e2e-autores-00a141-9bdf57677-9rxcm\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.227889 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:20:15.227869    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-00a141-serving-cert: secret "maas-api-e2e-autores-00a141-serving-cert" not found
Sep 16 12:20:15.227952 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:20:15.227939    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-maas-api-tls podName:244e6664-8f6f-4019-b8b7-bf86f29e474b nodeName:}" failed. No retries permitted until 2026-09-16 12:20:15.727921611 +0000 UTC m=+1012.481880321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-maas-api-tls") pod "maas-api-e2e-autores-00a141-9bdf57677-9rxcm" (UID: "244e6664-8f6f-4019-b8b7-bf86f29e474b") : secret "maas-api-e2e-autores-00a141-serving-cert" not found
Sep 16 12:20:15.228172 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.228153    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244e6664-8f6f-4019-b8b7-bf86f29e474b-openshift-service-ca\") pod \"maas-api-e2e-autores-00a141-9bdf57677-9rxcm\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.230191 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.230166    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-metrics-certs\") pod \"maas-api-e2e-autores-00a141-9bdf57677-9rxcm\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.238933 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.238906    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5h5b\" (UniqueName: \"kubernetes.io/projected/244e6664-8f6f-4019-b8b7-bf86f29e474b-kube-api-access-m5h5b\") pod \"maas-api-e2e-autores-00a141-9bdf57677-9rxcm\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.286701 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.286664    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:15.628145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.628113    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:15.716926 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.716898    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:15.732206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.732172    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-maas-api-tls\") pod \"maas-api-e2e-autores-00a141-9bdf57677-9rxcm\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.734830 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.734810    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-maas-api-tls\") pod \"maas-api-e2e-autores-00a141-9bdf57677-9rxcm\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.953494 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.953408    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:15.991667 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.991623    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"]
Sep 16 12:20:15.997862 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.997836    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:15.999983 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:15.999957    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5fb73413-serving-cert\""
Sep 16 12:20:16.004191 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.004167    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"]
Sep 16 12:20:16.031038 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.031006    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:16.034081 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.034054    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-maas-api-tls\") pod \"maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.034205 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.034100    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c844v\" (UniqueName: \"kubernetes.io/projected/2d0dc66b-6d5f-4a85-bb97-97678761ef18-kube-api-access-c844v\") pod \"maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.034205 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.034192    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-metrics-certs\") pod \"maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.034305 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.034285    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0dc66b-6d5f-4a85-bb97-97678761ef18-openshift-service-ca\") pod \"maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.131535 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.131302    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:16.135565 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.135460    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0dc66b-6d5f-4a85-bb97-97678761ef18-openshift-service-ca\") pod \"maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.135836 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.135762    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-maas-api-tls\") pod \"maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.135836 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.135814    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c844v\" (UniqueName: \"kubernetes.io/projected/2d0dc66b-6d5f-4a85-bb97-97678761ef18-kube-api-access-c844v\") pod \"maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.135983 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.135897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-metrics-certs\") pod \"maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.136039 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:20:16.135983    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-serving-cert: secret "maas-api-e2e-ait-5fb73413-serving-cert" not found
Sep 16 12:20:16.136093 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:20:16.136052    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-maas-api-tls podName:2d0dc66b-6d5f-4a85-bb97-97678761ef18 nodeName:}" failed. No retries permitted until 2026-09-16 12:20:16.636030359 +0000 UTC m=+1013.389989069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-maas-api-tls") pod "maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs" (UID: "2d0dc66b-6d5f-4a85-bb97-97678761ef18") : secret "maas-api-e2e-ait-5fb73413-serving-cert" not found
Sep 16 12:20:16.136576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.136531    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0dc66b-6d5f-4a85-bb97-97678761ef18-openshift-service-ca\") pod \"maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.138404 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.138382    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-metrics-certs\") pod \"maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.144922 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.144904    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c844v\" (UniqueName: \"kubernetes.io/projected/2d0dc66b-6d5f-4a85-bb97-97678761ef18-kube-api-access-c844v\") pod \"maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.236818 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.236358    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"]
Sep 16 12:20:16.239923 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:20:16.239899    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244e6664_8f6f_4019_b8b7_bf86f29e474b.slice/crio-6fdc2e3b83fed6225f8abc8348c1b6e5661f0d2fe3c08b6986e91ed9bb0c3439 WatchSource:0}: Error finding container 6fdc2e3b83fed6225f8abc8348c1b6e5661f0d2fe3c08b6986e91ed9bb0c3439: Status 404 returned error can't find the container with id 6fdc2e3b83fed6225f8abc8348c1b6e5661f0d2fe3c08b6986e91ed9bb0c3439
Sep 16 12:20:16.273487 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.273437    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:16.497442 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.497347    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:16.600457 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.600417    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:16.622915 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.622889    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"]
Sep 16 12:20:16.627582 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.627539    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"
Sep 16 12:20:16.630011 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.629985    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5fb73413\""
Sep 16 12:20:16.630121 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.630008    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5fb73413-dockercfg-w48zx\""
Sep 16 12:20:16.636530 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.635846    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"]
Sep 16 12:20:16.639971 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.639949    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfde1f9a-f6a9-42a2-b9ad-9cf543782e46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854\" (UID: \"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"
Sep 16 12:20:16.640086 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.640048    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-maas-api-tls\") pod \"maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.640086 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.640080    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cwdc\" (UniqueName: \"kubernetes.io/projected/bfde1f9a-f6a9-42a2-b9ad-9cf543782e46-kube-api-access-7cwdc\") pod \"payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854\" (UID: \"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"
Sep 16 12:20:16.642498 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.642476    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-maas-api-tls\") pod \"maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.740934 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.740891    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cwdc\" (UniqueName: \"kubernetes.io/projected/bfde1f9a-f6a9-42a2-b9ad-9cf543782e46-kube-api-access-7cwdc\") pod \"payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854\" (UID: \"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"
Sep 16 12:20:16.741093 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.740988    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfde1f9a-f6a9-42a2-b9ad-9cf543782e46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854\" (UID: \"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"
Sep 16 12:20:16.741769 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.741748    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfde1f9a-f6a9-42a2-b9ad-9cf543782e46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854\" (UID: \"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"
Sep 16 12:20:16.748734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.748665    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cwdc\" (UniqueName: \"kubernetes.io/projected/bfde1f9a-f6a9-42a2-b9ad-9cf543782e46-kube-api-access-7cwdc\") pod \"payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854\" (UID: \"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"
Sep 16 12:20:16.805562 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.805516    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:20:16.834242 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.834206    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm" event={"ID":"244e6664-8f6f-4019-b8b7-bf86f29e474b","Type":"ContainerStarted","Data":"3eb550c42feb91187e39884036d65ed40beb24271eb55442ad4a2e518a0b711d"}
Sep 16 12:20:16.834431 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.834273    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:16.834431 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.834296    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm" event={"ID":"244e6664-8f6f-4019-b8b7-bf86f29e474b","Type":"ContainerStarted","Data":"6fdc2e3b83fed6225f8abc8348c1b6e5661f0d2fe3c08b6986e91ed9bb0c3439"}
Sep 16 12:20:16.853419 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.853359    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm" podStartSLOduration=1.504908977 podStartE2EDuration="1.853336439s" totalImagesPullingTime="348.427462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:16.852008696 +0000 UTC m=+1013.605967429" watchObservedRunningTime="2026-09-16 12:20:16.853336439 +0000 UTC m=+1013.607295171"
Sep 16 12:20:16.915662 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.915626    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:16.939867 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:16.939839    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"
Sep 16 12:20:17.076698 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:17.076660    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:17.196145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:17.196115    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"]
Sep 16 12:20:17.197511 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:20:17.197483    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0dc66b_6d5f_4a85_bb97_97678761ef18.slice/crio-753d61e68ea534db135062dbe385130afefa769d45d219d4f04211a19b47b1a3 WatchSource:0}: Error finding container 753d61e68ea534db135062dbe385130afefa769d45d219d4f04211a19b47b1a3: Status 404 returned error can't find the container with id 753d61e68ea534db135062dbe385130afefa769d45d219d4f04211a19b47b1a3
Sep 16 12:20:17.226733 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:17.226710    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"]
Sep 16 12:20:17.228736 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:20:17.228709    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfde1f9a_f6a9_42a2_b9ad_9cf543782e46.slice/crio-f3ebd7fcd942632f7cf793f19d0c52df8caa23c38a93f6b62cd8ec2677f04428 WatchSource:0}: Error finding container f3ebd7fcd942632f7cf793f19d0c52df8caa23c38a93f6b62cd8ec2677f04428: Status 404 returned error can't find the container with id f3ebd7fcd942632f7cf793f19d0c52df8caa23c38a93f6b62cd8ec2677f04428
Sep 16 12:20:17.748173 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:17.748141    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:17.846675 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:17.846636    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854" event={"ID":"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46","Type":"ContainerStarted","Data":"fe6b1bb3f50997936e6391fb0934528c6bc49de187b0a69a65a5a55f7b7e0da7"}
Sep 16 12:20:17.846874 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:17.846727    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854" event={"ID":"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46","Type":"ContainerStarted","Data":"f3ebd7fcd942632f7cf793f19d0c52df8caa23c38a93f6b62cd8ec2677f04428"}
Sep 16 12:20:17.846874 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:17.846757    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"
Sep 16 12:20:17.848273 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:17.848244    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs" event={"ID":"2d0dc66b-6d5f-4a85-bb97-97678761ef18","Type":"ContainerStarted","Data":"febd5dbaa11b36cf82db89685c6ebae736e9c4a3eddf26c6fa51d0f3e1654026"}
Sep 16 12:20:17.848403 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:17.848295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs" event={"ID":"2d0dc66b-6d5f-4a85-bb97-97678761ef18","Type":"ContainerStarted","Data":"753d61e68ea534db135062dbe385130afefa769d45d219d4f04211a19b47b1a3"}
Sep 16 12:20:17.848403 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:17.848333    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:17.863258 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:17.863211    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854" podStartSLOduration=1.863197027 podStartE2EDuration="1.863197027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:17.861244706 +0000 UTC m=+1014.615203439" watchObservedRunningTime="2026-09-16 12:20:17.863197027 +0000 UTC m=+1014.617155758"
Sep 16 12:20:17.879450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:17.879410    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs" podStartSLOduration=2.550766033 podStartE2EDuration="2.879395147s" totalImagesPullingTime="328.629114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:17.877442654 +0000 UTC m=+1014.631401388" watchObservedRunningTime="2026-09-16 12:20:17.879395147 +0000 UTC m=+1014.633353878"
Sep 16 12:20:18.141517 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:18.141402    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:18.471924 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:18.471832    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:18.573500 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:18.573467    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:18.784787 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:18.784713    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"
Sep 16 12:20:18.791857 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:18.791827    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:18.926477 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:18.926450    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:19.349928 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:19.349887    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:20.101322 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:20.101286    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:21.157095 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:21.157042    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:21.532753 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:21.532659    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:21.636928 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:21.636891    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:21.794575 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:21.794458    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:21.899342 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:21.899303    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:21.981788 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:21.981762    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:22.081690 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:22.081587    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:22.453601 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:22.453557    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:22.856891 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:22.856860    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:20:23.323391 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:23.323312    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:23.867788 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:23.867752    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:23.990082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:23.990043    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:24.092933 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:24.092907    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:24.190503 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:24.190421    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:24.301337 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:24.301293    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:24.834681 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:24.834645    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:24.922631 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:24.922598    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:25.021805 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:25.021771    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:25.277310 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:25.277274    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:26.133487 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.133454    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:26.221596 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.221540    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"]
Sep 16 12:20:26.225267 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.225229    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:26.225433 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.225416    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.227558 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.227521    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ef3317-serving-cert\""
Sep 16 12:20:26.237064 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.237034    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"]
Sep 16 12:20:26.303356 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.303324    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:26.333217 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.333187    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.333372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.333284    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.333372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.333302    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrw7c\" (UniqueName: \"kubernetes.io/projected/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-kube-api-access-qrw7c\") pod \"maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.333372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.333324    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-metrics-certs\") pod \"maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.397239 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.397150    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:26.433965 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.433919    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.433965 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.433964    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrw7c\" (UniqueName: \"kubernetes.io/projected/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-kube-api-access-qrw7c\") pod \"maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.434224 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.434021    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-metrics-certs\") pod \"maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.434224 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.434100    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.434326 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:20:26.434268    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-serving-cert: secret "maas-api-e2e-inf-b-ef3317-serving-cert" not found
Sep 16 12:20:26.434365 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:20:26.434334    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-maas-api-tls podName:453f7dfa-4456-4a99-b2e3-6d4852d9d3ba nodeName:}" failed. No retries permitted until 2026-09-16 12:20:26.934318527 +0000 UTC m=+1023.688277249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-maas-api-tls") pod "maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c" (UID: "453f7dfa-4456-4a99-b2e3-6d4852d9d3ba") : secret "maas-api-e2e-inf-b-ef3317-serving-cert" not found
Sep 16 12:20:26.434796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.434766    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.436375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.436356    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-metrics-certs\") pod \"maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.441675 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.441652    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrw7c\" (UniqueName: \"kubernetes.io/projected/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-kube-api-access-qrw7c\") pod \"maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.496971 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.496935    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:26.600950 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.600909    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:26.840972 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.840946    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:26.939511 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.939479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.941853 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.941826    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:26.973534 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:26.973489    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:27.063735 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:27.063679    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:27.140389 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:27.140297    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:27.162933 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:27.162837    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:27.234870 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:27.234794    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:27.432566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:27.432527    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"]
Sep 16 12:20:27.434834 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:20:27.434808    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453f7dfa_4456_4a99_b2e3_6d4852d9d3ba.slice/crio-41be26cc7b499da36e1f7c3859d1b4c263086596166dfd7f593e978716761bb2 WatchSource:0}: Error finding container 41be26cc7b499da36e1f7c3859d1b4c263086596166dfd7f593e978716761bb2: Status 404 returned error can't find the container with id 41be26cc7b499da36e1f7c3859d1b4c263086596166dfd7f593e978716761bb2
Sep 16 12:20:27.444805 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:27.444775    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:27.905204 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:27.905171    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c" event={"ID":"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba","Type":"ContainerStarted","Data":"fd928d778461259ddc6b2ad5db79730cf4cac7d6f95c93d286982b58ef1acd7a"}
Sep 16 12:20:27.905385 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:27.905236    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c" event={"ID":"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba","Type":"ContainerStarted","Data":"41be26cc7b499da36e1f7c3859d1b4c263086596166dfd7f593e978716761bb2"}
Sep 16 12:20:27.905385 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:27.905270    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:27.922093 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:27.922034    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c" podStartSLOduration=1.619574104 podStartE2EDuration="1.922016302s" totalImagesPullingTime="302.442198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:27.919906137 +0000 UTC m=+1024.673864869" watchObservedRunningTime="2026-09-16 12:20:27.922016302 +0000 UTC m=+1024.675975089"
Sep 16 12:20:28.346366 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:28.346334    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:28.856770 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:28.856726    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"
Sep 16 12:20:29.132787 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:29.132696    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:29.767710 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:29.767680    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:30.562618 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:30.562493    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:30.924937 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:30.924903    2558 generic.go:356] "Generic (PLEG): container finished" podID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerID="e59e49b506d9b959ad6e7cdb45e9d562cb4a881719b4b195ea75eec32e8adde2" exitCode=0
Sep 16 12:20:30.925421 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:30.924986    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" event={"ID":"04a44004-4f18-4869-9290-faa4f6b4a9cf","Type":"ContainerDied","Data":"e59e49b506d9b959ad6e7cdb45e9d562cb4a881719b4b195ea75eec32e8adde2"}
Sep 16 12:20:30.925571 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:30.925533    2558 scope.go:117] "RemoveContainer" containerID="e59e49b506d9b959ad6e7cdb45e9d562cb4a881719b4b195ea75eec32e8adde2"
Sep 16 12:20:30.926811 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:30.926778    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:30.926920 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:30.926860    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:30.926920 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:30.926907    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:31.442082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:31.442031    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:31.656299 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:31.656197    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:31.931849 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:31.931816    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" event={"ID":"04a44004-4f18-4869-9290-faa4f6b4a9cf","Type":"ContainerStarted","Data":"3cb7aa51dd3739fbcf50270248796e7f174c7a41e2f4eb23f2b3ee29a7387032"}
Sep 16 12:20:32.151574 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:32.150641    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:32.151574 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:32.150770    2558 patch_prober.go:28] interesting pod/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 12:20:32.151574 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:32.150814    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 12:20:32.257432 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:32.257342    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:32.641298 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:32.641263    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:32.753336 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:32.753300    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:32.840481 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:32.840449    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:32.993883 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:32.993801    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:33.149629 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:33.149590    2558 patch_prober.go:28] interesting pod/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 12:20:33.149901 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:33.149873    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 12:20:33.242457 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:33.242420    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:33.919273 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:33.919241    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:20:34.149534 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:34.149492    2558 patch_prober.go:28] interesting pod/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 12:20:34.150161 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:34.150124    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 12:20:34.190619 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:34.190519    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:34.285186 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:34.285148    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:35.059040 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:35.058998    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:35.150240 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:35.150203    2558 patch_prober.go:28] interesting pod/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 12:20:35.150818 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:35.150784    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 12:20:35.750093 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:35.750056    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:35.762701 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:35.762669    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:36.150092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:36.150045    2558 patch_prober.go:28] interesting pod/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 12:20:36.150306 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:36.150128    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 12:20:36.512787 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:36.512704    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:37.149975 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:37.149937    2558 patch_prober.go:28] interesting pod/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 12:20:37.150164 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:37.150008    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 12:20:37.197991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:37.197918    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:38.069338 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:38.069301    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:38.153976 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:38.153945    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:38.154894 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:38.154879    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:20:38.767778 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:38.767746    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:38.824700 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:38.824664    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:38.922167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:38.922139    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:39.710470 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:39.710439    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:40.624693 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:40.624651    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:41.276054 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:41.276023    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:41.376230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:41.376199    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:41.997396 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:41.997363    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:42.730741 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:42.730707    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:43.421167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:43.421135    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:43.931841 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:43.931810    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"]
Sep 16 12:20:43.932050 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:43.932018    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs" podUID="2d0dc66b-6d5f-4a85-bb97-97678761ef18" containerName="maas-api" containerID="cri-o://febd5dbaa11b36cf82db89685c6ebae736e9c4a3eddf26c6fa51d0f3e1654026" gracePeriod=30
Sep 16 12:20:43.962137 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:43.962107    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:44.003948 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.003916    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"]
Sep 16 12:20:44.004129 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.004108    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854" podUID="bfde1f9a-f6a9-42a2-b9ad-9cf543782e46" containerName="payload-pre-processing" containerID="cri-o://fe6b1bb3f50997936e6391fb0934528c6bc49de187b0a69a65a5a55f7b7e0da7" gracePeriod=30
Sep 16 12:20:44.043560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.043511    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:44.105181 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.105101    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:44.163688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.163636    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:44.230539 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.230495    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:44.274270 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.274235    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:44.311946 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.311923    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"
Sep 16 12:20:44.331691 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.331661    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:44.421901 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.421866    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-metrics-certs\") pod \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") "
Sep 16 12:20:44.422343 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.421907    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cwdc\" (UniqueName: \"kubernetes.io/projected/bfde1f9a-f6a9-42a2-b9ad-9cf543782e46-kube-api-access-7cwdc\") pod \"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46\" (UID: \"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46\") "
Sep 16 12:20:44.422343 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.421942    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-maas-api-tls\") pod \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") "
Sep 16 12:20:44.422343 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.421965    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c844v\" (UniqueName: \"kubernetes.io/projected/2d0dc66b-6d5f-4a85-bb97-97678761ef18-kube-api-access-c844v\") pod \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") "
Sep 16 12:20:44.422343 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.422017    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfde1f9a-f6a9-42a2-b9ad-9cf543782e46-plugins-config-volume\") pod \"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46\" (UID: \"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46\") "
Sep 16 12:20:44.422343 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.422107    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0dc66b-6d5f-4a85-bb97-97678761ef18-openshift-service-ca\") pod \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\" (UID: \"2d0dc66b-6d5f-4a85-bb97-97678761ef18\") "
Sep 16 12:20:44.422625 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.422398    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfde1f9a-f6a9-42a2-b9ad-9cf543782e46-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bfde1f9a-f6a9-42a2-b9ad-9cf543782e46" (UID: "bfde1f9a-f6a9-42a2-b9ad-9cf543782e46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:20:44.422795 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.422761    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfde1f9a-f6a9-42a2-b9ad-9cf543782e46-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:20:44.424050 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.423999    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0dc66b-6d5f-4a85-bb97-97678761ef18-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2d0dc66b-6d5f-4a85-bb97-97678761ef18" (UID: "2d0dc66b-6d5f-4a85-bb97-97678761ef18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:20:44.424891 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.424838    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2d0dc66b-6d5f-4a85-bb97-97678761ef18" (UID: "2d0dc66b-6d5f-4a85-bb97-97678761ef18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:20:44.425011 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.424956    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2d0dc66b-6d5f-4a85-bb97-97678761ef18" (UID: "2d0dc66b-6d5f-4a85-bb97-97678761ef18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:20:44.425011 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.424987    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0dc66b-6d5f-4a85-bb97-97678761ef18-kube-api-access-c844v" (OuterVolumeSpecName: "kube-api-access-c844v") pod "2d0dc66b-6d5f-4a85-bb97-97678761ef18" (UID: "2d0dc66b-6d5f-4a85-bb97-97678761ef18"). InnerVolumeSpecName "kube-api-access-c844v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:44.425134 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.425008    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfde1f9a-f6a9-42a2-b9ad-9cf543782e46-kube-api-access-7cwdc" (OuterVolumeSpecName: "kube-api-access-7cwdc") pod "bfde1f9a-f6a9-42a2-b9ad-9cf543782e46" (UID: "bfde1f9a-f6a9-42a2-b9ad-9cf543782e46"). InnerVolumeSpecName "kube-api-access-7cwdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:44.524228 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.524144    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-metrics-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:20:44.524228 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.524173    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cwdc\" (UniqueName: \"kubernetes.io/projected/bfde1f9a-f6a9-42a2-b9ad-9cf543782e46-kube-api-access-7cwdc\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:20:44.524228 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.524184    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0dc66b-6d5f-4a85-bb97-97678761ef18-maas-api-tls\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:20:44.524228 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.524193    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c844v\" (UniqueName: \"kubernetes.io/projected/2d0dc66b-6d5f-4a85-bb97-97678761ef18-kube-api-access-c844v\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:20:44.524228 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.524202    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0dc66b-6d5f-4a85-bb97-97678761ef18-openshift-service-ca\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:20:44.639122 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.639083    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:44.749459 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.749403    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:44.848903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:44.848876    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:45.007154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.007122    2558 generic.go:356] "Generic (PLEG): container finished" podID="bfde1f9a-f6a9-42a2-b9ad-9cf543782e46" containerID="fe6b1bb3f50997936e6391fb0934528c6bc49de187b0a69a65a5a55f7b7e0da7" exitCode=0
Sep 16 12:20:45.007326 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.007181    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"
Sep 16 12:20:45.007326 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.007206    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854" event={"ID":"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46","Type":"ContainerDied","Data":"fe6b1bb3f50997936e6391fb0934528c6bc49de187b0a69a65a5a55f7b7e0da7"}
Sep 16 12:20:45.007326 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.007262    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854" event={"ID":"bfde1f9a-f6a9-42a2-b9ad-9cf543782e46","Type":"ContainerDied","Data":"f3ebd7fcd942632f7cf793f19d0c52df8caa23c38a93f6b62cd8ec2677f04428"}
Sep 16 12:20:45.007326 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.007285    2558 scope.go:117] "RemoveContainer" containerID="fe6b1bb3f50997936e6391fb0934528c6bc49de187b0a69a65a5a55f7b7e0da7"
Sep 16 12:20:45.008771 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.008692    2558 generic.go:356] "Generic (PLEG): container finished" podID="2d0dc66b-6d5f-4a85-bb97-97678761ef18" containerID="febd5dbaa11b36cf82db89685c6ebae736e9c4a3eddf26c6fa51d0f3e1654026" exitCode=0
Sep 16 12:20:45.008771 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.008734    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"
Sep 16 12:20:45.008771 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.008765    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs" event={"ID":"2d0dc66b-6d5f-4a85-bb97-97678761ef18","Type":"ContainerDied","Data":"febd5dbaa11b36cf82db89685c6ebae736e9c4a3eddf26c6fa51d0f3e1654026"}
Sep 16 12:20:45.008908 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.008806    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs" event={"ID":"2d0dc66b-6d5f-4a85-bb97-97678761ef18","Type":"ContainerDied","Data":"753d61e68ea534db135062dbe385130afefa769d45d219d4f04211a19b47b1a3"}
Sep 16 12:20:45.017181 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.017164    2558 scope.go:117] "RemoveContainer" containerID="fe6b1bb3f50997936e6391fb0934528c6bc49de187b0a69a65a5a55f7b7e0da7"
Sep 16 12:20:45.017478 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:20:45.017460    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6b1bb3f50997936e6391fb0934528c6bc49de187b0a69a65a5a55f7b7e0da7\": container with ID starting with fe6b1bb3f50997936e6391fb0934528c6bc49de187b0a69a65a5a55f7b7e0da7 not found: ID does not exist" containerID="fe6b1bb3f50997936e6391fb0934528c6bc49de187b0a69a65a5a55f7b7e0da7"
Sep 16 12:20:45.017593 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.017485    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6b1bb3f50997936e6391fb0934528c6bc49de187b0a69a65a5a55f7b7e0da7"} err="failed to get container status \"fe6b1bb3f50997936e6391fb0934528c6bc49de187b0a69a65a5a55f7b7e0da7\": rpc error: code = NotFound desc = could not find container \"fe6b1bb3f50997936e6391fb0934528c6bc49de187b0a69a65a5a55f7b7e0da7\": container with ID starting with fe6b1bb3f50997936e6391fb0934528c6bc49de187b0a69a65a5a55f7b7e0da7 not found: ID does not exist"
Sep 16 12:20:45.017593 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.017504    2558 scope.go:117] "RemoveContainer" containerID="febd5dbaa11b36cf82db89685c6ebae736e9c4a3eddf26c6fa51d0f3e1654026"
Sep 16 12:20:45.025398 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.025380    2558 scope.go:117] "RemoveContainer" containerID="febd5dbaa11b36cf82db89685c6ebae736e9c4a3eddf26c6fa51d0f3e1654026"
Sep 16 12:20:45.025784 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:20:45.025764    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"febd5dbaa11b36cf82db89685c6ebae736e9c4a3eddf26c6fa51d0f3e1654026\": container with ID starting with febd5dbaa11b36cf82db89685c6ebae736e9c4a3eddf26c6fa51d0f3e1654026 not found: ID does not exist" containerID="febd5dbaa11b36cf82db89685c6ebae736e9c4a3eddf26c6fa51d0f3e1654026"
Sep 16 12:20:45.025859 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.025789    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"febd5dbaa11b36cf82db89685c6ebae736e9c4a3eddf26c6fa51d0f3e1654026"} err="failed to get container status \"febd5dbaa11b36cf82db89685c6ebae736e9c4a3eddf26c6fa51d0f3e1654026\": rpc error: code = NotFound desc = could not find container \"febd5dbaa11b36cf82db89685c6ebae736e9c4a3eddf26c6fa51d0f3e1654026\": container with ID starting with febd5dbaa11b36cf82db89685c6ebae736e9c4a3eddf26c6fa51d0f3e1654026 not found: ID does not exist"
Sep 16 12:20:45.036885 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.036859    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"]
Sep 16 12:20:45.048012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.047990    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5fb73413-67fcc64454-vb854"]
Sep 16 12:20:45.071299 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.071269    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"]
Sep 16 12:20:45.092486 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.092375    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:45.094672 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.094648    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5fb73413-d48ff85f8-dxrcs"]
Sep 16 12:20:45.302718 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.302641    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"]
Sep 16 12:20:45.303074 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.303061    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d0dc66b-6d5f-4a85-bb97-97678761ef18" containerName="maas-api"
Sep 16 12:20:45.303122 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.303075    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d0dc66b-6d5f-4a85-bb97-97678761ef18" containerName="maas-api"
Sep 16 12:20:45.303122 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.303089    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfde1f9a-f6a9-42a2-b9ad-9cf543782e46" containerName="payload-pre-processing"
Sep 16 12:20:45.303122 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.303097    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfde1f9a-f6a9-42a2-b9ad-9cf543782e46" containerName="payload-pre-processing"
Sep 16 12:20:45.303222 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.303165    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfde1f9a-f6a9-42a2-b9ad-9cf543782e46" containerName="payload-pre-processing"
Sep 16 12:20:45.303222 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.303174    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d0dc66b-6d5f-4a85-bb97-97678761ef18" containerName="maas-api"
Sep 16 12:20:45.307769 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.307743    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.311826 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.311799    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4c69c87e-gw-openshift-default-dockercfg-bvkf6\""
Sep 16 12:20:45.319155 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.319133    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"]
Sep 16 12:20:45.328092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.328072    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:45.432206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.432175    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beff105f-94de-4da6-b552-a9d126a4da15-istiod-ca-cert\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.432206 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.432210    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-credential-socket\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.432673 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.432237    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6zxm\" (UniqueName: \"kubernetes.io/projected/beff105f-94de-4da6-b552-a9d126a4da15-kube-api-access-z6zxm\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.432673 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.432322    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-workload-certs\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.432673 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.432361    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beff105f-94de-4da6-b552-a9d126a4da15-istio-token\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.432673 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.432431    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beff105f-94de-4da6-b552-a9d126a4da15-istio-podinfo\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.432673 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.432459    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-workload-socket\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.432673 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.432478    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-istio-data\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.432673 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.432498    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-istio-envoy\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.518381 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.518346    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:45.533699 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.533668    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beff105f-94de-4da6-b552-a9d126a4da15-istiod-ca-cert\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.533699 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.533701    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-credential-socket\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.533893 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.533720    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6zxm\" (UniqueName: \"kubernetes.io/projected/beff105f-94de-4da6-b552-a9d126a4da15-kube-api-access-z6zxm\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.533893 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.533838    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-workload-certs\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.533893 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.533879    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beff105f-94de-4da6-b552-a9d126a4da15-istio-token\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.534053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.533942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beff105f-94de-4da6-b552-a9d126a4da15-istio-podinfo\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.534053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.533970    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-workload-socket\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.534053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.533992    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-istio-data\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.534053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.534020    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-istio-envoy\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.534267 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.534246    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-credential-socket\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.534337 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.534317    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-workload-certs\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.534414 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.534389    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-workload-socket\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.534465 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.534431    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-istio-data\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.534633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.534617    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beff105f-94de-4da6-b552-a9d126a4da15-istiod-ca-cert\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.536328 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.536307    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-istio-envoy\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.536493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.536473    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beff105f-94de-4da6-b552-a9d126a4da15-istio-podinfo\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.541212 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.541191    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beff105f-94de-4da6-b552-a9d126a4da15-istio-token\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.541413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.541392    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6zxm\" (UniqueName: \"kubernetes.io/projected/beff105f-94de-4da6-b552-a9d126a4da15-kube-api-access-z6zxm\") pod \"e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") " pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.619531 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.619500    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:45.622120 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.622100    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:45.757788 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.757754    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d0dc66b-6d5f-4a85-bb97-97678761ef18" path="/var/lib/kubelet/pods/2d0dc66b-6d5f-4a85-bb97-97678761ef18/volumes"
Sep 16 12:20:45.758162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.758147    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfde1f9a-f6a9-42a2-b9ad-9cf543782e46" path="/var/lib/kubelet/pods/bfde1f9a-f6a9-42a2-b9ad-9cf543782e46/volumes"
Sep 16 12:20:45.902628 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.902604    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"]
Sep 16 12:20:45.904982 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:20:45.904954    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeff105f_94de_4da6_b552_a9d126a4da15.slice/crio-62f84f14498d707b97bc30d4e2330cd0d78877fd067c8b7882d559f57d088620 WatchSource:0}: Error finding container 62f84f14498d707b97bc30d4e2330cd0d78877fd067c8b7882d559f57d088620: Status 404 returned error can't find the container with id 62f84f14498d707b97bc30d4e2330cd0d78877fd067c8b7882d559f57d088620
Sep 16 12:20:45.907297 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.907265    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:45.907375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.907327    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:45.907375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:45.907354    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:46.019034 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:46.018997    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" event={"ID":"beff105f-94de-4da6-b552-a9d126a4da15","Type":"ContainerStarted","Data":"fba523465afab9443e317cf534cf4dfabe6e626410a808363d7ccdca5feaa2cb"}
Sep 16 12:20:46.019222 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:46.019056    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" event={"ID":"beff105f-94de-4da6-b552-a9d126a4da15","Type":"ContainerStarted","Data":"62f84f14498d707b97bc30d4e2330cd0d78877fd067c8b7882d559f57d088620"}
Sep 16 12:20:46.041261 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:46.041196    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" podStartSLOduration=1.04117381 podStartE2EDuration="1.04117381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:46.037366213 +0000 UTC m=+1042.791324945" watchObservedRunningTime="2026-09-16 12:20:46.04117381 +0000 UTC m=+1042.795132544"
Sep 16 12:20:46.403125 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:46.403085    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:46.620619 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:46.620573    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:46.625934 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:46.625910    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:47.024352 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:47.024323    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:47.025224 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:47.025204    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:20:47.207008 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:47.206970    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:47.578588 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:47.578557    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:47.678773 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:47.678747    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:48.109648 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:48.109621    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:48.739106 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:48.739072    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:49.478625 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:49.478595    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:49.584779 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:49.584739    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:49.953572 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:49.953514    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:50.039702 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.039670    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:50.041053 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.041032    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"]
Sep 16 12:20:50.044735 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.044718    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"
Sep 16 12:20:50.046918 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.046898    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4c69c87e\""
Sep 16 12:20:50.053993 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.053972    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"]
Sep 16 12:20:50.092513 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.092478    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"]
Sep 16 12:20:50.096017 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.096001    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"
Sep 16 12:20:50.098358 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.098336    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4c69c87e-dockercfg-69pg7\""
Sep 16 12:20:50.106693 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.106666    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"]
Sep 16 12:20:50.178743 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.178710    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b276aec2-2924-4291-902f-3a616a56397d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp\" (UID: \"b276aec2-2924-4291-902f-3a616a56397d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"
Sep 16 12:20:50.178941 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.178823    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6p2z\" (UniqueName: \"kubernetes.io/projected/b276aec2-2924-4291-902f-3a616a56397d-kube-api-access-w6p2z\") pod \"payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp\" (UID: \"b276aec2-2924-4291-902f-3a616a56397d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"
Sep 16 12:20:50.178941 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.178856    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77zz2\" (UniqueName: \"kubernetes.io/projected/40575894-1ee9-4570-a58e-4296846a0cf5-kube-api-access-77zz2\") pod \"payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6\" (UID: \"40575894-1ee9-4570-a58e-4296846a0cf5\") " pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"
Sep 16 12:20:50.178941 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.178912    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40575894-1ee9-4570-a58e-4296846a0cf5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6\" (UID: \"40575894-1ee9-4570-a58e-4296846a0cf5\") " pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"
Sep 16 12:20:50.279921 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.279828    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6p2z\" (UniqueName: \"kubernetes.io/projected/b276aec2-2924-4291-902f-3a616a56397d-kube-api-access-w6p2z\") pod \"payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp\" (UID: \"b276aec2-2924-4291-902f-3a616a56397d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"
Sep 16 12:20:50.279921 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.279873    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77zz2\" (UniqueName: \"kubernetes.io/projected/40575894-1ee9-4570-a58e-4296846a0cf5-kube-api-access-77zz2\") pod \"payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6\" (UID: \"40575894-1ee9-4570-a58e-4296846a0cf5\") " pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"
Sep 16 12:20:50.279921 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.279916    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40575894-1ee9-4570-a58e-4296846a0cf5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6\" (UID: \"40575894-1ee9-4570-a58e-4296846a0cf5\") " pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"
Sep 16 12:20:50.280205 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.279939    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b276aec2-2924-4291-902f-3a616a56397d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp\" (UID: \"b276aec2-2924-4291-902f-3a616a56397d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"
Sep 16 12:20:50.280681 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.280653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b276aec2-2924-4291-902f-3a616a56397d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp\" (UID: \"b276aec2-2924-4291-902f-3a616a56397d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"
Sep 16 12:20:50.280794 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.280712    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40575894-1ee9-4570-a58e-4296846a0cf5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6\" (UID: \"40575894-1ee9-4570-a58e-4296846a0cf5\") " pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"
Sep 16 12:20:50.288017 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.287993    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77zz2\" (UniqueName: \"kubernetes.io/projected/40575894-1ee9-4570-a58e-4296846a0cf5-kube-api-access-77zz2\") pod \"payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6\" (UID: \"40575894-1ee9-4570-a58e-4296846a0cf5\") " pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"
Sep 16 12:20:50.288145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.288128    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6p2z\" (UniqueName: \"kubernetes.io/projected/b276aec2-2924-4291-902f-3a616a56397d-kube-api-access-w6p2z\") pod \"payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp\" (UID: \"b276aec2-2924-4291-902f-3a616a56397d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"
Sep 16 12:20:50.357098 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.357053    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"
Sep 16 12:20:50.406232 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.406171    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"
Sep 16 12:20:50.441774 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.441725    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:50.632613 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.632586    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"]
Sep 16 12:20:50.635320 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:20:50.635292    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb276aec2_2924_4291_902f_3a616a56397d.slice/crio-0ade7fd37441e100dcdefc09057822d9e0fd536029715e0c56266eafb24daf1a WatchSource:0}: Error finding container 0ade7fd37441e100dcdefc09057822d9e0fd536029715e0c56266eafb24daf1a: Status 404 returned error can't find the container with id 0ade7fd37441e100dcdefc09057822d9e0fd536029715e0c56266eafb24daf1a
Sep 16 12:20:50.687606 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:50.687580    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"]
Sep 16 12:20:50.688629 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:20:50.688606    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40575894_1ee9_4570_a58e_4296846a0cf5.slice/crio-7076f4af303f6723935f48200c53259ceb4423ccaa0f512a9a7b52c1710c555a WatchSource:0}: Error finding container 7076f4af303f6723935f48200c53259ceb4423ccaa0f512a9a7b52c1710c555a: Status 404 returned error can't find the container with id 7076f4af303f6723935f48200c53259ceb4423ccaa0f512a9a7b52c1710c555a
Sep 16 12:20:51.046955 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:51.046845    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6" event={"ID":"40575894-1ee9-4570-a58e-4296846a0cf5","Type":"ContainerStarted","Data":"57cd85c5ecaf18aaf60f7f26b4af882a71f59b1dae615701107dd775431c147a"}
Sep 16 12:20:51.046955 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:51.046911    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6" event={"ID":"40575894-1ee9-4570-a58e-4296846a0cf5","Type":"ContainerStarted","Data":"7076f4af303f6723935f48200c53259ceb4423ccaa0f512a9a7b52c1710c555a"}
Sep 16 12:20:51.047436 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:51.046960    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"
Sep 16 12:20:51.048656 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:51.048633    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp" event={"ID":"b276aec2-2924-4291-902f-3a616a56397d","Type":"ContainerStarted","Data":"e65b6a353053bc21049961b82c71a2dc18da618791b4d1b9738ec6890aae3cc3"}
Sep 16 12:20:51.048791 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:51.048669    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp" event={"ID":"b276aec2-2924-4291-902f-3a616a56397d","Type":"ContainerStarted","Data":"0ade7fd37441e100dcdefc09057822d9e0fd536029715e0c56266eafb24daf1a"}
Sep 16 12:20:51.048791 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:51.048766    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"
Sep 16 12:20:51.069571 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:51.069511    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6" podStartSLOduration=1.069498572 podStartE2EDuration="1.069498572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:51.066684246 +0000 UTC m=+1047.820642978" watchObservedRunningTime="2026-09-16 12:20:51.069498572 +0000 UTC m=+1047.823457303"
Sep 16 12:20:51.083855 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:51.083802    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp" podStartSLOduration=1.083783608 podStartE2EDuration="1.083783608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:51.08061823 +0000 UTC m=+1047.834576966" watchObservedRunningTime="2026-09-16 12:20:51.083783608 +0000 UTC m=+1047.837742345"
Sep 16 12:20:51.149802 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:51.149759    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:51.829897 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:51.829865    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:51.955702 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:51.955668    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:52.542906 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:52.542877    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:54.224290 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:20:54.224260    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:02.057781 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:02.057738    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"
Sep 16 12:21:02.058843 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:02.058814    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"
Sep 16 12:21:08.144341 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:08.144298    2558 generic.go:356] "Generic (PLEG): container finished" podID="beff105f-94de-4da6-b552-a9d126a4da15" containerID="fba523465afab9443e317cf534cf4dfabe6e626410a808363d7ccdca5feaa2cb" exitCode=0
Sep 16 12:21:08.144858 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:08.144368    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" event={"ID":"beff105f-94de-4da6-b552-a9d126a4da15","Type":"ContainerDied","Data":"fba523465afab9443e317cf534cf4dfabe6e626410a808363d7ccdca5feaa2cb"}
Sep 16 12:21:08.144858 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:08.144792    2558 scope.go:117] "RemoveContainer" containerID="fba523465afab9443e317cf534cf4dfabe6e626410a808363d7ccdca5feaa2cb"
Sep 16 12:21:08.145885 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:08.145848    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:21:08.145975 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:08.145913    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:21:08.145975 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:08.145945    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:21:09.155406 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:09.155359    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" event={"ID":"beff105f-94de-4da6-b552-a9d126a4da15","Type":"ContainerStarted","Data":"5dc7b5d3f2c7908f3b51f953482f3844195e1f9fdd3239a07b1528eb8d39236a"}
Sep 16 12:21:09.620417 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:09.620377    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:21:09.620698 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:09.620671    2558 patch_prober.go:28] interesting pod/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:21:09.620756 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:09.620729    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" podUID="beff105f-94de-4da6-b552-a9d126a4da15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 12:21:10.620526 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:10.620479    2558 patch_prober.go:28] interesting pod/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:21:10.620956 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:10.620573    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" podUID="beff105f-94de-4da6-b552-a9d126a4da15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 12:21:11.620230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:11.620190    2558 patch_prober.go:28] interesting pod/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:21:11.620407 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:11.620252    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" podUID="beff105f-94de-4da6-b552-a9d126a4da15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 12:21:11.745912 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:11.745877    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:12.482796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:12.482752    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:12.620659 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:12.620619    2558 patch_prober.go:28] interesting pod/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:21:12.620867 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:12.620694    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" podUID="beff105f-94de-4da6-b552-a9d126a4da15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 12:21:13.228490 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:13.228452    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:13.619943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:13.619903    2558 patch_prober.go:28] interesting pod/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:21:13.620118 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:13.619959    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" podUID="beff105f-94de-4da6-b552-a9d126a4da15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 12:21:13.953347 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:13.953248    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:14.624643 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:14.624614    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:21:14.625039 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:14.624893    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:21:14.625495 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:14.625476    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:21:14.629201 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:14.629178    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:15.263627 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:15.263591    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:15.916535 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:15.916499    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:16.737732 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:16.737692    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:17.365155 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:17.365118    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:24.122315 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:24.122286    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:28.216332 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:28.216299    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:28.758603 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:28.758541    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:28.769902 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:28.769861    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"]
Sep 16 12:21:28.770178 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:28.770130    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6" podUID="40575894-1ee9-4570-a58e-4296846a0cf5" containerName="payload-processing" containerID="cri-o://57cd85c5ecaf18aaf60f7f26b4af882a71f59b1dae615701107dd775431c147a" gracePeriod=30
Sep 16 12:21:28.774045 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:28.774014    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"]
Sep 16 12:21:28.774295 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:28.774270    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp" podUID="b276aec2-2924-4291-902f-3a616a56397d" containerName="payload-pre-processing" containerID="cri-o://e65b6a353053bc21049961b82c71a2dc18da618791b4d1b9738ec6890aae3cc3" gracePeriod=30
Sep 16 12:21:28.856064 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:28.856038    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:28.921993 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:28.921926    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:29.036283 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.036185    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"]
Sep 16 12:21:29.037181 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.037133    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" podUID="beff105f-94de-4da6-b552-a9d126a4da15" containerName="istio-proxy" containerID="cri-o://5dc7b5d3f2c7908f3b51f953482f3844195e1f9fdd3239a07b1528eb8d39236a" gracePeriod=30
Sep 16 12:21:29.061151 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.061119    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:29.075777 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.075742    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"
Sep 16 12:21:29.103978 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.103955    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"
Sep 16 12:21:29.165335 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.165303    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6p2z\" (UniqueName: \"kubernetes.io/projected/b276aec2-2924-4291-902f-3a616a56397d-kube-api-access-w6p2z\") pod \"b276aec2-2924-4291-902f-3a616a56397d\" (UID: \"b276aec2-2924-4291-902f-3a616a56397d\") "
Sep 16 12:21:29.165523 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.165493    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b276aec2-2924-4291-902f-3a616a56397d-plugins-config-volume\") pod \"b276aec2-2924-4291-902f-3a616a56397d\" (UID: \"b276aec2-2924-4291-902f-3a616a56397d\") "
Sep 16 12:21:29.165842 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.165819    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b276aec2-2924-4291-902f-3a616a56397d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b276aec2-2924-4291-902f-3a616a56397d" (UID: "b276aec2-2924-4291-902f-3a616a56397d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:21:29.167668 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.167640    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b276aec2-2924-4291-902f-3a616a56397d-kube-api-access-w6p2z" (OuterVolumeSpecName: "kube-api-access-w6p2z") pod "b276aec2-2924-4291-902f-3a616a56397d" (UID: "b276aec2-2924-4291-902f-3a616a56397d"). InnerVolumeSpecName "kube-api-access-w6p2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:21:29.265630 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.265590    2558 generic.go:356] "Generic (PLEG): container finished" podID="40575894-1ee9-4570-a58e-4296846a0cf5" containerID="57cd85c5ecaf18aaf60f7f26b4af882a71f59b1dae615701107dd775431c147a" exitCode=0
Sep 16 12:21:29.266092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.265641    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"
Sep 16 12:21:29.266092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.265675    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6" event={"ID":"40575894-1ee9-4570-a58e-4296846a0cf5","Type":"ContainerDied","Data":"57cd85c5ecaf18aaf60f7f26b4af882a71f59b1dae615701107dd775431c147a"}
Sep 16 12:21:29.266092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.265734    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6" event={"ID":"40575894-1ee9-4570-a58e-4296846a0cf5","Type":"ContainerDied","Data":"7076f4af303f6723935f48200c53259ceb4423ccaa0f512a9a7b52c1710c555a"}
Sep 16 12:21:29.266092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.265765    2558 scope.go:117] "RemoveContainer" containerID="57cd85c5ecaf18aaf60f7f26b4af882a71f59b1dae615701107dd775431c147a"
Sep 16 12:21:29.266092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.266048    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77zz2\" (UniqueName: \"kubernetes.io/projected/40575894-1ee9-4570-a58e-4296846a0cf5-kube-api-access-77zz2\") pod \"40575894-1ee9-4570-a58e-4296846a0cf5\" (UID: \"40575894-1ee9-4570-a58e-4296846a0cf5\") "
Sep 16 12:21:29.266378 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.266220    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40575894-1ee9-4570-a58e-4296846a0cf5-plugins-config-volume\") pod \"40575894-1ee9-4570-a58e-4296846a0cf5\" (UID: \"40575894-1ee9-4570-a58e-4296846a0cf5\") "
Sep 16 12:21:29.266538 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.266513    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b276aec2-2924-4291-902f-3a616a56397d-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.266687 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.266571    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6p2z\" (UniqueName: \"kubernetes.io/projected/b276aec2-2924-4291-902f-3a616a56397d-kube-api-access-w6p2z\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.266754 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.266686    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40575894-1ee9-4570-a58e-4296846a0cf5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40575894-1ee9-4570-a58e-4296846a0cf5" (UID: "40575894-1ee9-4570-a58e-4296846a0cf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:21:29.267371 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.267344    2558 generic.go:356] "Generic (PLEG): container finished" podID="b276aec2-2924-4291-902f-3a616a56397d" containerID="e65b6a353053bc21049961b82c71a2dc18da618791b4d1b9738ec6890aae3cc3" exitCode=0
Sep 16 12:21:29.267470 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.267387    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"
Sep 16 12:21:29.267470 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.267392    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp" event={"ID":"b276aec2-2924-4291-902f-3a616a56397d","Type":"ContainerDied","Data":"e65b6a353053bc21049961b82c71a2dc18da618791b4d1b9738ec6890aae3cc3"}
Sep 16 12:21:29.267470 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.267447    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp" event={"ID":"b276aec2-2924-4291-902f-3a616a56397d","Type":"ContainerDied","Data":"0ade7fd37441e100dcdefc09057822d9e0fd536029715e0c56266eafb24daf1a"}
Sep 16 12:21:29.268489 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.268467    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40575894-1ee9-4570-a58e-4296846a0cf5-kube-api-access-77zz2" (OuterVolumeSpecName: "kube-api-access-77zz2") pod "40575894-1ee9-4570-a58e-4296846a0cf5" (UID: "40575894-1ee9-4570-a58e-4296846a0cf5"). InnerVolumeSpecName "kube-api-access-77zz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:21:29.278688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.278664    2558 scope.go:117] "RemoveContainer" containerID="57cd85c5ecaf18aaf60f7f26b4af882a71f59b1dae615701107dd775431c147a"
Sep 16 12:21:29.278995 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:21:29.278975    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57cd85c5ecaf18aaf60f7f26b4af882a71f59b1dae615701107dd775431c147a\": container with ID starting with 57cd85c5ecaf18aaf60f7f26b4af882a71f59b1dae615701107dd775431c147a not found: ID does not exist" containerID="57cd85c5ecaf18aaf60f7f26b4af882a71f59b1dae615701107dd775431c147a"
Sep 16 12:21:29.279065 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.279002    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57cd85c5ecaf18aaf60f7f26b4af882a71f59b1dae615701107dd775431c147a"} err="failed to get container status \"57cd85c5ecaf18aaf60f7f26b4af882a71f59b1dae615701107dd775431c147a\": rpc error: code = NotFound desc = could not find container \"57cd85c5ecaf18aaf60f7f26b4af882a71f59b1dae615701107dd775431c147a\": container with ID starting with 57cd85c5ecaf18aaf60f7f26b4af882a71f59b1dae615701107dd775431c147a not found: ID does not exist"
Sep 16 12:21:29.279065 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.279020    2558 scope.go:117] "RemoveContainer" containerID="e65b6a353053bc21049961b82c71a2dc18da618791b4d1b9738ec6890aae3cc3"
Sep 16 12:21:29.288111 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.287862    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"]
Sep 16 12:21:29.288111 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.287982    2558 scope.go:117] "RemoveContainer" containerID="e65b6a353053bc21049961b82c71a2dc18da618791b4d1b9738ec6890aae3cc3"
Sep 16 12:21:29.288263 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:21:29.288234    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65b6a353053bc21049961b82c71a2dc18da618791b4d1b9738ec6890aae3cc3\": container with ID starting with e65b6a353053bc21049961b82c71a2dc18da618791b4d1b9738ec6890aae3cc3 not found: ID does not exist" containerID="e65b6a353053bc21049961b82c71a2dc18da618791b4d1b9738ec6890aae3cc3"
Sep 16 12:21:29.288406 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.288274    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65b6a353053bc21049961b82c71a2dc18da618791b4d1b9738ec6890aae3cc3"} err="failed to get container status \"e65b6a353053bc21049961b82c71a2dc18da618791b4d1b9738ec6890aae3cc3\": rpc error: code = NotFound desc = could not find container \"e65b6a353053bc21049961b82c71a2dc18da618791b4d1b9738ec6890aae3cc3\": container with ID starting with e65b6a353053bc21049961b82c71a2dc18da618791b4d1b9738ec6890aae3cc3 not found: ID does not exist"
Sep 16 12:21:29.291402 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.291382    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4c69c87e-b747d5d66-ghrtp"]
Sep 16 12:21:29.332661 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.332635    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:29.367665 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.367637    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40575894-1ee9-4570-a58e-4296846a0cf5-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.367665 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.367661    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77zz2\" (UniqueName: \"kubernetes.io/projected/40575894-1ee9-4570-a58e-4296846a0cf5-kube-api-access-77zz2\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.508127 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.508096    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:29.587602 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.587567    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"]
Sep 16 12:21:29.589366 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.589343    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4c69c87e-7fbfc76566-jskc6"]
Sep 16 12:21:29.627622 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.627576    2558 patch_prober.go:28] interesting pod/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 16 12:21:29.627794 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.627649    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" podUID="beff105f-94de-4da6-b552-a9d126a4da15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:21:29.758271 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.758237    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40575894-1ee9-4570-a58e-4296846a0cf5" path="/var/lib/kubelet/pods/40575894-1ee9-4570-a58e-4296846a0cf5/volumes"
Sep 16 12:21:29.758638 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.758624    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b276aec2-2924-4291-902f-3a616a56397d" path="/var/lib/kubelet/pods/b276aec2-2924-4291-902f-3a616a56397d/volumes"
Sep 16 12:21:29.807732 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.807704    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:29.905736 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:29.905703    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:30.120779 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:30.120745    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:31.025028 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:31.024993    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:31.898496 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:31.898453    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:32.622327 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:32.622294    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:32.717021 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:32.716988    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:33.393681 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:33.393650    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:34.160376 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.160354    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:34.261350 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.261323    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:34.295601 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.295571    2558 generic.go:356] "Generic (PLEG): container finished" podID="beff105f-94de-4da6-b552-a9d126a4da15" containerID="5dc7b5d3f2c7908f3b51f953482f3844195e1f9fdd3239a07b1528eb8d39236a" exitCode=0
Sep 16 12:21:34.295712 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.295571    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" event={"ID":"beff105f-94de-4da6-b552-a9d126a4da15","Type":"ContainerDied","Data":"5dc7b5d3f2c7908f3b51f953482f3844195e1f9fdd3239a07b1528eb8d39236a"}
Sep 16 12:21:34.295712 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.295708    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k" event={"ID":"beff105f-94de-4da6-b552-a9d126a4da15","Type":"ContainerDied","Data":"62f84f14498d707b97bc30d4e2330cd0d78877fd067c8b7882d559f57d088620"}
Sep 16 12:21:34.295787 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.295724    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62f84f14498d707b97bc30d4e2330cd0d78877fd067c8b7882d559f57d088620"
Sep 16 12:21:34.295787 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.295738    2558 scope.go:117] "RemoveContainer" containerID="fba523465afab9443e317cf534cf4dfabe6e626410a808363d7ccdca5feaa2cb"
Sep 16 12:21:34.304911 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.304891    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:21:34.361645 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.361623    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:34.416635 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.416611    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beff105f-94de-4da6-b552-a9d126a4da15-istio-token\") pod \"beff105f-94de-4da6-b552-a9d126a4da15\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") "
Sep 16 12:21:34.416796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.416644    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-istio-envoy\") pod \"beff105f-94de-4da6-b552-a9d126a4da15\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") "
Sep 16 12:21:34.416796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.416668    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-workload-certs\") pod \"beff105f-94de-4da6-b552-a9d126a4da15\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") "
Sep 16 12:21:34.416796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.416694    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6zxm\" (UniqueName: \"kubernetes.io/projected/beff105f-94de-4da6-b552-a9d126a4da15-kube-api-access-z6zxm\") pod \"beff105f-94de-4da6-b552-a9d126a4da15\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") "
Sep 16 12:21:34.416796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.416711    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beff105f-94de-4da6-b552-a9d126a4da15-istio-podinfo\") pod \"beff105f-94de-4da6-b552-a9d126a4da15\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") "
Sep 16 12:21:34.416796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.416766    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-istio-data\") pod \"beff105f-94de-4da6-b552-a9d126a4da15\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") "
Sep 16 12:21:34.416796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.416792    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beff105f-94de-4da6-b552-a9d126a4da15-istiod-ca-cert\") pod \"beff105f-94de-4da6-b552-a9d126a4da15\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") "
Sep 16 12:21:34.417087 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.416820    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-credential-socket\") pod \"beff105f-94de-4da6-b552-a9d126a4da15\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") "
Sep 16 12:21:34.417087 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.416887    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-workload-socket\") pod \"beff105f-94de-4da6-b552-a9d126a4da15\" (UID: \"beff105f-94de-4da6-b552-a9d126a4da15\") "
Sep 16 12:21:34.417087 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.416959    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "beff105f-94de-4da6-b552-a9d126a4da15" (UID: "beff105f-94de-4da6-b552-a9d126a4da15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:34.417251 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.417153    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "beff105f-94de-4da6-b552-a9d126a4da15" (UID: "beff105f-94de-4da6-b552-a9d126a4da15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:34.417251 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.417238    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-credential-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:21:34.417354 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.417259    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-workload-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:21:34.417354 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.417339    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-istio-data" (OuterVolumeSpecName: "istio-data") pod "beff105f-94de-4da6-b552-a9d126a4da15" (UID: "beff105f-94de-4da6-b552-a9d126a4da15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:34.417461 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.417356    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "beff105f-94de-4da6-b552-a9d126a4da15" (UID: "beff105f-94de-4da6-b552-a9d126a4da15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:34.417461 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.417361    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beff105f-94de-4da6-b552-a9d126a4da15-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "beff105f-94de-4da6-b552-a9d126a4da15" (UID: "beff105f-94de-4da6-b552-a9d126a4da15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:21:34.418950 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.418916    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "beff105f-94de-4da6-b552-a9d126a4da15" (UID: "beff105f-94de-4da6-b552-a9d126a4da15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:34.419066 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.418971    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beff105f-94de-4da6-b552-a9d126a4da15-istio-token" (OuterVolumeSpecName: "istio-token") pod "beff105f-94de-4da6-b552-a9d126a4da15" (UID: "beff105f-94de-4da6-b552-a9d126a4da15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:21:34.419279 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.419256    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beff105f-94de-4da6-b552-a9d126a4da15-kube-api-access-z6zxm" (OuterVolumeSpecName: "kube-api-access-z6zxm") pod "beff105f-94de-4da6-b552-a9d126a4da15" (UID: "beff105f-94de-4da6-b552-a9d126a4da15"). InnerVolumeSpecName "kube-api-access-z6zxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:21:34.419387 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.419318    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beff105f-94de-4da6-b552-a9d126a4da15-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "beff105f-94de-4da6-b552-a9d126a4da15" (UID: "beff105f-94de-4da6-b552-a9d126a4da15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:21:34.518726 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.518692    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beff105f-94de-4da6-b552-a9d126a4da15-istio-token\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:21:34.518726 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.518719    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-istio-envoy\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:21:34.518726 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.518729    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6zxm\" (UniqueName: \"kubernetes.io/projected/beff105f-94de-4da6-b552-a9d126a4da15-kube-api-access-z6zxm\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:21:34.518960 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.518738    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beff105f-94de-4da6-b552-a9d126a4da15-istio-podinfo\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:21:34.518960 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.518747    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-istio-data\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:21:34.518960 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.518774    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beff105f-94de-4da6-b552-a9d126a4da15-istiod-ca-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:21:34.518960 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.518784    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beff105f-94de-4da6-b552-a9d126a4da15-workload-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:21:34.649936 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.649847    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:34.749537 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.749500    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"]
Sep 16 12:21:34.749950 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.749935    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beff105f-94de-4da6-b552-a9d126a4da15" containerName="istio-proxy"
Sep 16 12:21:34.749995 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.749952    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="beff105f-94de-4da6-b552-a9d126a4da15" containerName="istio-proxy"
Sep 16 12:21:34.749995 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.749964    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b276aec2-2924-4291-902f-3a616a56397d" containerName="payload-pre-processing"
Sep 16 12:21:34.749995 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.749969    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b276aec2-2924-4291-902f-3a616a56397d" containerName="payload-pre-processing"
Sep 16 12:21:34.749995 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.749977    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40575894-1ee9-4570-a58e-4296846a0cf5" containerName="payload-processing"
Sep 16 12:21:34.749995 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.749983    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="40575894-1ee9-4570-a58e-4296846a0cf5" containerName="payload-processing"
Sep 16 12:21:34.749995 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.749991    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beff105f-94de-4da6-b552-a9d126a4da15" containerName="istio-proxy"
Sep 16 12:21:34.749995 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.749996    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="beff105f-94de-4da6-b552-a9d126a4da15" containerName="istio-proxy"
Sep 16 12:21:34.750204 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.750058    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="40575894-1ee9-4570-a58e-4296846a0cf5" containerName="payload-processing"
Sep 16 12:21:34.750204 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.750065    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="beff105f-94de-4da6-b552-a9d126a4da15" containerName="istio-proxy"
Sep 16 12:21:34.750204 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.750072    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b276aec2-2924-4291-902f-3a616a56397d" containerName="payload-pre-processing"
Sep 16 12:21:34.751474 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.751445    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:34.754718 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.754693    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"
Sep 16 12:21:34.757071 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.757050    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1f9d3de8-dockercfg-sq67d\""
Sep 16 12:21:34.757174 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.757096    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1f9d3de8\""
Sep 16 12:21:34.763198 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.763173    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"]
Sep 16 12:21:34.766760 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.766713    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"]
Sep 16 12:21:34.767202 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.767183    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="beff105f-94de-4da6-b552-a9d126a4da15" containerName="istio-proxy"
Sep 16 12:21:34.770358 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.770339    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"
Sep 16 12:21:34.784477 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.784455    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"]
Sep 16 12:21:34.922303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.922200    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e88d1e-e557-4422-baa5-40eab38731b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br\" (UID: \"45e88d1e-e557-4422-baa5-40eab38731b9\") " pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"
Sep 16 12:21:34.922303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.922253    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed57ab38-ff06-40aa-ac3e-59cc3e073b70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v\" (UID: \"ed57ab38-ff06-40aa-ac3e-59cc3e073b70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"
Sep 16 12:21:34.922571 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.922310    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfm94\" (UniqueName: \"kubernetes.io/projected/45e88d1e-e557-4422-baa5-40eab38731b9-kube-api-access-sfm94\") pod \"payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br\" (UID: \"45e88d1e-e557-4422-baa5-40eab38731b9\") " pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"
Sep 16 12:21:34.922571 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:34.922338    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9ldw\" (UniqueName: \"kubernetes.io/projected/ed57ab38-ff06-40aa-ac3e-59cc3e073b70-kube-api-access-s9ldw\") pod \"payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v\" (UID: \"ed57ab38-ff06-40aa-ac3e-59cc3e073b70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"
Sep 16 12:21:35.023392 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.023352    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e88d1e-e557-4422-baa5-40eab38731b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br\" (UID: \"45e88d1e-e557-4422-baa5-40eab38731b9\") " pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"
Sep 16 12:21:35.023594 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.023395    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed57ab38-ff06-40aa-ac3e-59cc3e073b70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v\" (UID: \"ed57ab38-ff06-40aa-ac3e-59cc3e073b70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"
Sep 16 12:21:35.023594 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.023445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfm94\" (UniqueName: \"kubernetes.io/projected/45e88d1e-e557-4422-baa5-40eab38731b9-kube-api-access-sfm94\") pod \"payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br\" (UID: \"45e88d1e-e557-4422-baa5-40eab38731b9\") " pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"
Sep 16 12:21:35.023594 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.023476    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9ldw\" (UniqueName: \"kubernetes.io/projected/ed57ab38-ff06-40aa-ac3e-59cc3e073b70-kube-api-access-s9ldw\") pod \"payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v\" (UID: \"ed57ab38-ff06-40aa-ac3e-59cc3e073b70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"
Sep 16 12:21:35.024233 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.024213    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e88d1e-e557-4422-baa5-40eab38731b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br\" (UID: \"45e88d1e-e557-4422-baa5-40eab38731b9\") " pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"
Sep 16 12:21:35.024316 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.024214    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed57ab38-ff06-40aa-ac3e-59cc3e073b70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v\" (UID: \"ed57ab38-ff06-40aa-ac3e-59cc3e073b70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"
Sep 16 12:21:35.055123 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.055099    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9ldw\" (UniqueName: \"kubernetes.io/projected/ed57ab38-ff06-40aa-ac3e-59cc3e073b70-kube-api-access-s9ldw\") pod \"payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v\" (UID: \"ed57ab38-ff06-40aa-ac3e-59cc3e073b70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"
Sep 16 12:21:35.055245 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.055199    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfm94\" (UniqueName: \"kubernetes.io/projected/45e88d1e-e557-4422-baa5-40eab38731b9-kube-api-access-sfm94\") pod \"payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br\" (UID: \"45e88d1e-e557-4422-baa5-40eab38731b9\") " pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"
Sep 16 12:21:35.067141 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.067119    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"
Sep 16 12:21:35.080365 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.080339    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"
Sep 16 12:21:35.091452 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.091423    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:35.302092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.301997    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"
Sep 16 12:21:35.338515 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.338479    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"]
Sep 16 12:21:35.340586 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.340562    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4c69c87e-gw-openshift-default-58fdd84f9b-bm59k"]
Sep 16 12:21:35.349834 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.349813    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"]
Sep 16 12:21:35.350713 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:21:35.350692    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded57ab38_ff06_40aa_ac3e_59cc3e073b70.slice/crio-bd2363ec0fa9e8bdebdd7da7d02b28b4fa9f0985f6fd0a14145e04ec0e10ffc2 WatchSource:0}: Error finding container bd2363ec0fa9e8bdebdd7da7d02b28b4fa9f0985f6fd0a14145e04ec0e10ffc2: Status 404 returned error can't find the container with id bd2363ec0fa9e8bdebdd7da7d02b28b4fa9f0985f6fd0a14145e04ec0e10ffc2
Sep 16 12:21:35.364266 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.364235    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"]
Sep 16 12:21:35.365227 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:21:35.365203    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e88d1e_e557_4422_baa5_40eab38731b9.slice/crio-3b9bacb716366623247c6d674e3cdcbccbc8b5621bea85996c5d61b6de2c277c WatchSource:0}: Error finding container 3b9bacb716366623247c6d674e3cdcbccbc8b5621bea85996c5d61b6de2c277c: Status 404 returned error can't find the container with id 3b9bacb716366623247c6d674e3cdcbccbc8b5621bea85996c5d61b6de2c277c
Sep 16 12:21:35.750377 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.750341    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:35.766452 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:35.766415    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beff105f-94de-4da6-b552-a9d126a4da15" path="/var/lib/kubelet/pods/beff105f-94de-4da6-b552-a9d126a4da15/volumes"
Sep 16 12:21:36.307985 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:36.307947    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br" event={"ID":"45e88d1e-e557-4422-baa5-40eab38731b9","Type":"ContainerStarted","Data":"ebcee0e1a90a80e83af89a2a3f21656b7c33638ba6d9ac8eba3f3d71da81c9d3"}
Sep 16 12:21:36.308450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:36.308010    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br" event={"ID":"45e88d1e-e557-4422-baa5-40eab38731b9","Type":"ContainerStarted","Data":"3b9bacb716366623247c6d674e3cdcbccbc8b5621bea85996c5d61b6de2c277c"}
Sep 16 12:21:36.308450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:36.308041    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"
Sep 16 12:21:36.309613 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:36.309590    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v" event={"ID":"ed57ab38-ff06-40aa-ac3e-59cc3e073b70","Type":"ContainerStarted","Data":"ad3c7c67b165461b1ef9f2cd431fc8b6e63d7bc2bc23c901f0f78012e11c482d"}
Sep 16 12:21:36.309726 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:36.309628    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v" event={"ID":"ed57ab38-ff06-40aa-ac3e-59cc3e073b70","Type":"ContainerStarted","Data":"bd2363ec0fa9e8bdebdd7da7d02b28b4fa9f0985f6fd0a14145e04ec0e10ffc2"}
Sep 16 12:21:36.309726 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:36.309680    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"
Sep 16 12:21:36.324161 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:36.324112    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br" podStartSLOduration=2.324098968 podStartE2EDuration="2.324098968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:36.321958476 +0000 UTC m=+1093.075917206" watchObservedRunningTime="2026-09-16 12:21:36.324098968 +0000 UTC m=+1093.078057699"
Sep 16 12:21:36.336800 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:36.336742    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v" podStartSLOduration=2.336727308 podStartE2EDuration="2.336727308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:36.335959376 +0000 UTC m=+1093.089918108" watchObservedRunningTime="2026-09-16 12:21:36.336727308 +0000 UTC m=+1093.090686037"
Sep 16 12:21:36.543407 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:36.543372    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:37.254831 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:37.254799    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:46.728802 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.728765    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"]
Sep 16 12:21:46.733728 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.733706    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.736003 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.735981    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-00a141\"/\"openshift-service-ca.crt\""
Sep 16 12:21:46.736098 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.736009    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-00a141\"/\"default-dockercfg-jpx8n\""
Sep 16 12:21:46.736098 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.736034    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-00a141\"/\"autores-model-00a141-kserve-self-signed-certs\""
Sep 16 12:21:46.736098 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.735987    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-00a141\"/\"kube-root-ca.crt\""
Sep 16 12:21:46.742995 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.742970    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"]
Sep 16 12:21:46.832147 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.832108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-dshm\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.832332 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.832219    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-model-cache\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.832332 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.832257    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-tmp-dir\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.832470 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.832346    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a94984-baa6-488b-b3ef-55a375dd6a67-tls-certs\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.832470 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.832383    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-home\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.832470 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.832412    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmwwv\" (UniqueName: \"kubernetes.io/projected/a2a94984-baa6-488b-b3ef-55a375dd6a67-kube-api-access-cmwwv\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.839602 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.839532    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:46.933201 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.933161    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-tmp-dir\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.933384 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.933226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a94984-baa6-488b-b3ef-55a375dd6a67-tls-certs\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.933384 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.933256    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-home\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.933384 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.933278    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmwwv\" (UniqueName: \"kubernetes.io/projected/a2a94984-baa6-488b-b3ef-55a375dd6a67-kube-api-access-cmwwv\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.933384 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.933314    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-dshm\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.933601 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.933404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-model-cache\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.933659 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.933637    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-tmp-dir\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.933701 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.933666    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-home\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.933738 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.933725    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-model-cache\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.935615 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.935596    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-dshm\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.935818 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.935802    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a94984-baa6-488b-b3ef-55a375dd6a67-tls-certs\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:46.941635 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:46.941611    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmwwv\" (UniqueName: \"kubernetes.io/projected/a2a94984-baa6-488b-b3ef-55a375dd6a67-kube-api-access-cmwwv\") pod \"autores-model-00a141-kserve-cc86989d7-4kbwm\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") " pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:47.044587 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:47.044489    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:47.313614 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:47.313589    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"]
Sep 16 12:21:47.315748 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:21:47.315717    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2a94984_baa6_488b_b3ef_55a375dd6a67.slice/crio-5a9cac5cd5aeac9a5eaa30acdee8d26ab351416113961ad75bb4ddf43df6f256 WatchSource:0}: Error finding container 5a9cac5cd5aeac9a5eaa30acdee8d26ab351416113961ad75bb4ddf43df6f256: Status 404 returned error can't find the container with id 5a9cac5cd5aeac9a5eaa30acdee8d26ab351416113961ad75bb4ddf43df6f256
Sep 16 12:21:47.316364 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:47.316339    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"
Sep 16 12:21:47.316466 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:47.316403    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"
Sep 16 12:21:47.369229 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:47.369202    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm" event={"ID":"a2a94984-baa6-488b-b3ef-55a375dd6a67","Type":"ContainerStarted","Data":"5a9cac5cd5aeac9a5eaa30acdee8d26ab351416113961ad75bb4ddf43df6f256"}
Sep 16 12:21:47.399919 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:47.399886    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:47.452857 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:47.452756    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:48.376558 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:48.376504    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm" event={"ID":"a2a94984-baa6-488b-b3ef-55a375dd6a67","Type":"ContainerStarted","Data":"b561dd724482dfa979b2073dcb58f354f6ef3ac1c6512c6fb92d8200ea9d191e"}
Sep 16 12:21:48.393837 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:48.393791    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm" podStartSLOduration=2.393775651 podStartE2EDuration="2.393775651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:48.391760956 +0000 UTC m=+1105.145719688" watchObservedRunningTime="2026-09-16 12:21:48.393775651 +0000 UTC m=+1105.147734382"
Sep 16 12:21:56.226060 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:56.226025    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:56.847215 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:56.847185    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:57.045641 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:57.045596    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:57.045902 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:57.045869    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:57.058229 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:57.058206    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:57.433987 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:57.433956    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:21:57.650200 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:57.650173    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:58.335071 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:58.335037    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:59.230196 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:59.230160    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:59.853002 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:21:59.852970    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:00.430030 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.429994    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"]
Sep 16 12:22:00.433822 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.433805    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.439655 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.438485    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f79e22\"/\"default-dockercfg-nk66n\""
Sep 16 12:22:00.439655 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.438761    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f79e22\"/\"kube-root-ca.crt\""
Sep 16 12:22:00.439655 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.439077    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f79e22\"/\"openshift-service-ca.crt\""
Sep 16 12:22:00.439655 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.439298    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f79e22\"/\"test-model-f79e22-kserve-self-signed-certs\""
Sep 16 12:22:00.447844 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.447317    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"]
Sep 16 12:22:00.456684 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.456650    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:00.566476 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.566449    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172b8fc8-9fac-40c1-8900-543728756262-tls-certs\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.566685 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.566523    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd4fx\" (UniqueName: \"kubernetes.io/projected/172b8fc8-9fac-40c1-8900-543728756262-kube-api-access-qd4fx\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.566685 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.566568    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-dshm\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.566685 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.566593    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-model-cache\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.566685 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.566657    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-home\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.566824 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.566703    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-tmp-dir\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.617857 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.617825    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:00.667941 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.667906    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd4fx\" (UniqueName: \"kubernetes.io/projected/172b8fc8-9fac-40c1-8900-543728756262-kube-api-access-qd4fx\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.667941 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.667942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-dshm\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.668173 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.668068    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-model-cache\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.668173 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.668111    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-home\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.668173 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.668143    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-tmp-dir\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.668330 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.668193    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172b8fc8-9fac-40c1-8900-543728756262-tls-certs\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.668484 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.668464    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-model-cache\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.668577 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.668530    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-home\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.668626 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.668576    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-tmp-dir\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.670151 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.670123    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-dshm\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.670504 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.670488    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172b8fc8-9fac-40c1-8900-543728756262-tls-certs\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.675362 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.675342    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd4fx\" (UniqueName: \"kubernetes.io/projected/172b8fc8-9fac-40c1-8900-543728756262-kube-api-access-qd4fx\") pod \"test-model-f79e22-kserve-56555cd7ff-hw9qm\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") " pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:00.757022 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:00.756930    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:01.028162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:01.028138    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"]
Sep 16 12:22:01.030176 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:22:01.030139    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod172b8fc8_9fac_40c1_8900_543728756262.slice/crio-5b4495f7c8700dc76fe22016fc808771ce8a534dd01d281784d3531efe4f2b08 WatchSource:0}: Error finding container 5b4495f7c8700dc76fe22016fc808771ce8a534dd01d281784d3531efe4f2b08: Status 404 returned error can't find the container with id 5b4495f7c8700dc76fe22016fc808771ce8a534dd01d281784d3531efe4f2b08
Sep 16 12:22:01.250763 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:01.250721    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:01.458224 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:01.458187    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm" event={"ID":"172b8fc8-9fac-40c1-8900-543728756262","Type":"ContainerStarted","Data":"2d58cb0e752bf0776c57da7c31f42dc231480fede430813880c8c6b6d2cbfb7a"}
Sep 16 12:22:01.458659 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:01.458232    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm" event={"ID":"172b8fc8-9fac-40c1-8900-543728756262","Type":"ContainerStarted","Data":"5b4495f7c8700dc76fe22016fc808771ce8a534dd01d281784d3531efe4f2b08"}
Sep 16 12:22:01.473897 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:01.473847    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm" podStartSLOduration=1.473830368 podStartE2EDuration="1.473830368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:22:01.472790288 +0000 UTC m=+1118.226749020" watchObservedRunningTime="2026-09-16 12:22:01.473830368 +0000 UTC m=+1118.227789099"
Sep 16 12:22:01.919344 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:01.919305    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:02.641822 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:02.641790    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:03.307885 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:03.307776    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:04.024734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:04.024700    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:04.607193 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:04.607163    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:04.686624 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:04.686590    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"]
Sep 16 12:22:04.686859 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:04.686831    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br" podUID="45e88d1e-e557-4422-baa5-40eab38731b9" containerName="payload-processing" containerID="cri-o://ebcee0e1a90a80e83af89a2a3f21656b7c33638ba6d9ac8eba3f3d71da81c9d3" gracePeriod=30
Sep 16 12:22:04.699103 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:04.699065    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"]
Sep 16 12:22:04.699438 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:04.699407    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v" podUID="ed57ab38-ff06-40aa-ac3e-59cc3e073b70" containerName="payload-pre-processing" containerID="cri-o://ad3c7c67b165461b1ef9f2cd431fc8b6e63d7bc2bc23c901f0f78012e11c482d" gracePeriod=30
Sep 16 12:22:04.721155 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:04.721121    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:04.822701 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:04.822669    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:04.895815 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:04.895745    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:04.970465 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:04.969420    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"
Sep 16 12:22:05.006465 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.006432    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"
Sep 16 12:22:05.114502 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.114462    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfm94\" (UniqueName: \"kubernetes.io/projected/45e88d1e-e557-4422-baa5-40eab38731b9-kube-api-access-sfm94\") pod \"45e88d1e-e557-4422-baa5-40eab38731b9\" (UID: \"45e88d1e-e557-4422-baa5-40eab38731b9\") "
Sep 16 12:22:05.115014 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.114526    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e88d1e-e557-4422-baa5-40eab38731b9-plugins-config-volume\") pod \"45e88d1e-e557-4422-baa5-40eab38731b9\" (UID: \"45e88d1e-e557-4422-baa5-40eab38731b9\") "
Sep 16 12:22:05.115014 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.114615    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed57ab38-ff06-40aa-ac3e-59cc3e073b70-plugins-config-volume\") pod \"ed57ab38-ff06-40aa-ac3e-59cc3e073b70\" (UID: \"ed57ab38-ff06-40aa-ac3e-59cc3e073b70\") "
Sep 16 12:22:05.115014 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.114652    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9ldw\" (UniqueName: \"kubernetes.io/projected/ed57ab38-ff06-40aa-ac3e-59cc3e073b70-kube-api-access-s9ldw\") pod \"ed57ab38-ff06-40aa-ac3e-59cc3e073b70\" (UID: \"ed57ab38-ff06-40aa-ac3e-59cc3e073b70\") "
Sep 16 12:22:05.115014 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.114951    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed57ab38-ff06-40aa-ac3e-59cc3e073b70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed57ab38-ff06-40aa-ac3e-59cc3e073b70" (UID: "ed57ab38-ff06-40aa-ac3e-59cc3e073b70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:05.115014 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.114976    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e88d1e-e557-4422-baa5-40eab38731b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45e88d1e-e557-4422-baa5-40eab38731b9" (UID: "45e88d1e-e557-4422-baa5-40eab38731b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:05.116577 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.116556    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed57ab38-ff06-40aa-ac3e-59cc3e073b70-kube-api-access-s9ldw" (OuterVolumeSpecName: "kube-api-access-s9ldw") pod "ed57ab38-ff06-40aa-ac3e-59cc3e073b70" (UID: "ed57ab38-ff06-40aa-ac3e-59cc3e073b70"). InnerVolumeSpecName "kube-api-access-s9ldw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:05.116825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.116808    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e88d1e-e557-4422-baa5-40eab38731b9-kube-api-access-sfm94" (OuterVolumeSpecName: "kube-api-access-sfm94") pod "45e88d1e-e557-4422-baa5-40eab38731b9" (UID: "45e88d1e-e557-4422-baa5-40eab38731b9"). InnerVolumeSpecName "kube-api-access-sfm94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:05.216112 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.216024    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e88d1e-e557-4422-baa5-40eab38731b9-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:05.216112 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.216053    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed57ab38-ff06-40aa-ac3e-59cc3e073b70-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:05.216112 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.216062    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9ldw\" (UniqueName: \"kubernetes.io/projected/ed57ab38-ff06-40aa-ac3e-59cc3e073b70-kube-api-access-s9ldw\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:05.216112 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.216072    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfm94\" (UniqueName: \"kubernetes.io/projected/45e88d1e-e557-4422-baa5-40eab38731b9-kube-api-access-sfm94\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:05.265031 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.264988    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:05.479395 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.479300    2558 generic.go:356] "Generic (PLEG): container finished" podID="45e88d1e-e557-4422-baa5-40eab38731b9" containerID="ebcee0e1a90a80e83af89a2a3f21656b7c33638ba6d9ac8eba3f3d71da81c9d3" exitCode=0
Sep 16 12:22:05.479395 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.479350    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"
Sep 16 12:22:05.479625 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.479394    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br" event={"ID":"45e88d1e-e557-4422-baa5-40eab38731b9","Type":"ContainerDied","Data":"ebcee0e1a90a80e83af89a2a3f21656b7c33638ba6d9ac8eba3f3d71da81c9d3"}
Sep 16 12:22:05.479625 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.479458    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br" event={"ID":"45e88d1e-e557-4422-baa5-40eab38731b9","Type":"ContainerDied","Data":"3b9bacb716366623247c6d674e3cdcbccbc8b5621bea85996c5d61b6de2c277c"}
Sep 16 12:22:05.479625 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.479490    2558 scope.go:117] "RemoveContainer" containerID="ebcee0e1a90a80e83af89a2a3f21656b7c33638ba6d9ac8eba3f3d71da81c9d3"
Sep 16 12:22:05.480853 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.480784    2558 generic.go:356] "Generic (PLEG): container finished" podID="ed57ab38-ff06-40aa-ac3e-59cc3e073b70" containerID="ad3c7c67b165461b1ef9f2cd431fc8b6e63d7bc2bc23c901f0f78012e11c482d" exitCode=0
Sep 16 12:22:05.480930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.480847    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"
Sep 16 12:22:05.481004 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.480856    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v" event={"ID":"ed57ab38-ff06-40aa-ac3e-59cc3e073b70","Type":"ContainerDied","Data":"ad3c7c67b165461b1ef9f2cd431fc8b6e63d7bc2bc23c901f0f78012e11c482d"}
Sep 16 12:22:05.481004 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.480974    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v" event={"ID":"ed57ab38-ff06-40aa-ac3e-59cc3e073b70","Type":"ContainerDied","Data":"bd2363ec0fa9e8bdebdd7da7d02b28b4fa9f0985f6fd0a14145e04ec0e10ffc2"}
Sep 16 12:22:05.489707 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.489684    2558 scope.go:117] "RemoveContainer" containerID="ebcee0e1a90a80e83af89a2a3f21656b7c33638ba6d9ac8eba3f3d71da81c9d3"
Sep 16 12:22:05.490038 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:22:05.490016    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebcee0e1a90a80e83af89a2a3f21656b7c33638ba6d9ac8eba3f3d71da81c9d3\": container with ID starting with ebcee0e1a90a80e83af89a2a3f21656b7c33638ba6d9ac8eba3f3d71da81c9d3 not found: ID does not exist" containerID="ebcee0e1a90a80e83af89a2a3f21656b7c33638ba6d9ac8eba3f3d71da81c9d3"
Sep 16 12:22:05.490143 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.490057    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebcee0e1a90a80e83af89a2a3f21656b7c33638ba6d9ac8eba3f3d71da81c9d3"} err="failed to get container status \"ebcee0e1a90a80e83af89a2a3f21656b7c33638ba6d9ac8eba3f3d71da81c9d3\": rpc error: code = NotFound desc = could not find container \"ebcee0e1a90a80e83af89a2a3f21656b7c33638ba6d9ac8eba3f3d71da81c9d3\": container with ID starting with ebcee0e1a90a80e83af89a2a3f21656b7c33638ba6d9ac8eba3f3d71da81c9d3 not found: ID does not exist"
Sep 16 12:22:05.490143 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.490081    2558 scope.go:117] "RemoveContainer" containerID="ad3c7c67b165461b1ef9f2cd431fc8b6e63d7bc2bc23c901f0f78012e11c482d"
Sep 16 12:22:05.498653 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.498632    2558 scope.go:117] "RemoveContainer" containerID="ad3c7c67b165461b1ef9f2cd431fc8b6e63d7bc2bc23c901f0f78012e11c482d"
Sep 16 12:22:05.498935 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:22:05.498916    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3c7c67b165461b1ef9f2cd431fc8b6e63d7bc2bc23c901f0f78012e11c482d\": container with ID starting with ad3c7c67b165461b1ef9f2cd431fc8b6e63d7bc2bc23c901f0f78012e11c482d not found: ID does not exist" containerID="ad3c7c67b165461b1ef9f2cd431fc8b6e63d7bc2bc23c901f0f78012e11c482d"
Sep 16 12:22:05.498991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.498941    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3c7c67b165461b1ef9f2cd431fc8b6e63d7bc2bc23c901f0f78012e11c482d"} err="failed to get container status \"ad3c7c67b165461b1ef9f2cd431fc8b6e63d7bc2bc23c901f0f78012e11c482d\": rpc error: code = NotFound desc = could not find container \"ad3c7c67b165461b1ef9f2cd431fc8b6e63d7bc2bc23c901f0f78012e11c482d\": container with ID starting with ad3c7c67b165461b1ef9f2cd431fc8b6e63d7bc2bc23c901f0f78012e11c482d not found: ID does not exist"
Sep 16 12:22:05.503726 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.503704    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"]
Sep 16 12:22:05.506519 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.506499    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1f9d3de8-6689dc949d-dbh9v"]
Sep 16 12:22:05.516136 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.516114    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"]
Sep 16 12:22:05.520115 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.520096    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1f9d3de8-c676b46c-ct4br"]
Sep 16 12:22:05.639293 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.639256    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:05.758356 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.758276    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45e88d1e-e557-4422-baa5-40eab38731b9" path="/var/lib/kubelet/pods/45e88d1e-e557-4422-baa5-40eab38731b9/volumes"
Sep 16 12:22:05.758752 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.758735    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed57ab38-ff06-40aa-ac3e-59cc3e073b70" path="/var/lib/kubelet/pods/ed57ab38-ff06-40aa-ac3e-59cc3e073b70/volumes"
Sep 16 12:22:05.963397 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:05.963350    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:06.063482 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:06.063387    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:06.391685 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:06.391655    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:06.635327 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:06.635294    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:06.733212 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:06.733136    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:06.941958 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:06.941923    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:07.423355 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:07.423304    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:08.100791 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:08.100755    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:08.835093 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:08.835060    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:08.931728 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:08.931692    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:09.767801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:09.767750    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:10.556425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:10.556354    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:10.757097 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:10.757061    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:10.757294 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:10.757121    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:10.769970 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:10.769949    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:11.200848 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:11.200815    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:11.525843 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:11.525750    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:22:12.036893 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:12.036839    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:12.670665 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:12.670636    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:12.770897 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:12.770850    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:13.054177 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.054091    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:13.177219 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.177187    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:13.260481 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.260437    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:13.265587 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.265563    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"]
Sep 16 12:22:13.265964 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.265953    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed57ab38-ff06-40aa-ac3e-59cc3e073b70" containerName="payload-pre-processing"
Sep 16 12:22:13.266008 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.265967    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed57ab38-ff06-40aa-ac3e-59cc3e073b70" containerName="payload-pre-processing"
Sep 16 12:22:13.266008 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.265982    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e88d1e-e557-4422-baa5-40eab38731b9" containerName="payload-processing"
Sep 16 12:22:13.266008 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.265988    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e88d1e-e557-4422-baa5-40eab38731b9" containerName="payload-processing"
Sep 16 12:22:13.266096 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.266054    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e88d1e-e557-4422-baa5-40eab38731b9" containerName="payload-processing"
Sep 16 12:22:13.266096 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.266061    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed57ab38-ff06-40aa-ac3e-59cc3e073b70" containerName="payload-pre-processing"
Sep 16 12:22:13.270337 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.270322    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"
Sep 16 12:22:13.272769 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.272741    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c2aae110\""
Sep 16 12:22:13.272881 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.272862    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c2aae110-dockercfg-qzhnh\""
Sep 16 12:22:13.276711 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.276683    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"]
Sep 16 12:22:13.304706 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.304623    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"]
Sep 16 12:22:13.308136 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.308120    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"
Sep 16 12:22:13.319073 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.319049    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"]
Sep 16 12:22:13.400734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.400691    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84958338-ef8c-42cf-ac40-5e37fce76d38-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88\" (UID: \"84958338-ef8c-42cf-ac40-5e37fce76d38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"
Sep 16 12:22:13.401172 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.401136    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plr9d\" (UniqueName: \"kubernetes.io/projected/84958338-ef8c-42cf-ac40-5e37fce76d38-kube-api-access-plr9d\") pod \"payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88\" (UID: \"84958338-ef8c-42cf-ac40-5e37fce76d38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"
Sep 16 12:22:13.502245 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.502209    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tjwf\" (UniqueName: \"kubernetes.io/projected/bd0f5528-e83a-47c0-afde-db26417e5f2a-kube-api-access-5tjwf\") pod \"payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw\" (UID: \"bd0f5528-e83a-47c0-afde-db26417e5f2a\") " pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"
Sep 16 12:22:13.502245 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.502247    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0f5528-e83a-47c0-afde-db26417e5f2a-plugins-config-volume\") pod \"payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw\" (UID: \"bd0f5528-e83a-47c0-afde-db26417e5f2a\") " pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"
Sep 16 12:22:13.502431 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.502274    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84958338-ef8c-42cf-ac40-5e37fce76d38-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88\" (UID: \"84958338-ef8c-42cf-ac40-5e37fce76d38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"
Sep 16 12:22:13.502431 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.502354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plr9d\" (UniqueName: \"kubernetes.io/projected/84958338-ef8c-42cf-ac40-5e37fce76d38-kube-api-access-plr9d\") pod \"payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88\" (UID: \"84958338-ef8c-42cf-ac40-5e37fce76d38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"
Sep 16 12:22:13.502904 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.502885    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84958338-ef8c-42cf-ac40-5e37fce76d38-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88\" (UID: \"84958338-ef8c-42cf-ac40-5e37fce76d38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"
Sep 16 12:22:13.510099 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.510074    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plr9d\" (UniqueName: \"kubernetes.io/projected/84958338-ef8c-42cf-ac40-5e37fce76d38-kube-api-access-plr9d\") pod \"payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88\" (UID: \"84958338-ef8c-42cf-ac40-5e37fce76d38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"
Sep 16 12:22:13.582814 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.582728    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"
Sep 16 12:22:13.603470 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.603439    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tjwf\" (UniqueName: \"kubernetes.io/projected/bd0f5528-e83a-47c0-afde-db26417e5f2a-kube-api-access-5tjwf\") pod \"payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw\" (UID: \"bd0f5528-e83a-47c0-afde-db26417e5f2a\") " pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"
Sep 16 12:22:13.603617 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.603480    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0f5528-e83a-47c0-afde-db26417e5f2a-plugins-config-volume\") pod \"payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw\" (UID: \"bd0f5528-e83a-47c0-afde-db26417e5f2a\") " pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"
Sep 16 12:22:13.604421 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.604394    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0f5528-e83a-47c0-afde-db26417e5f2a-plugins-config-volume\") pod \"payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw\" (UID: \"bd0f5528-e83a-47c0-afde-db26417e5f2a\") " pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"
Sep 16 12:22:13.614627 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.614594    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tjwf\" (UniqueName: \"kubernetes.io/projected/bd0f5528-e83a-47c0-afde-db26417e5f2a-kube-api-access-5tjwf\") pod \"payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw\" (UID: \"bd0f5528-e83a-47c0-afde-db26417e5f2a\") " pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"
Sep 16 12:22:13.619110 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.619088    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"
Sep 16 12:22:13.858413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.858385    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"]
Sep 16 12:22:13.860014 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:22:13.859984    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84958338_ef8c_42cf_ac40_5e37fce76d38.slice/crio-ebbc23fafb9abd0a5769b08c9e668639b08bc8162359da94cdb5fb328c3bff44 WatchSource:0}: Error finding container ebbc23fafb9abd0a5769b08c9e668639b08bc8162359da94cdb5fb328c3bff44: Status 404 returned error can't find the container with id ebbc23fafb9abd0a5769b08c9e668639b08bc8162359da94cdb5fb328c3bff44
Sep 16 12:22:13.902573 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:13.902438    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"]
Sep 16 12:22:13.903875 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:22:13.903854    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0f5528_e83a_47c0_afde_db26417e5f2a.slice/crio-ce529114f7544e142a1a32b935ab2a5407240c4ba029d6694d7cdd587ffda330 WatchSource:0}: Error finding container ce529114f7544e142a1a32b935ab2a5407240c4ba029d6694d7cdd587ffda330: Status 404 returned error can't find the container with id ce529114f7544e142a1a32b935ab2a5407240c4ba029d6694d7cdd587ffda330
Sep 16 12:22:14.068377 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:14.068340    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:14.530720 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:14.530686    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88" event={"ID":"84958338-ef8c-42cf-ac40-5e37fce76d38","Type":"ContainerStarted","Data":"8edf28bf9ba1526ee2d3b3d43918e1e31a598f8a747425cb93f2cd99e65f70bb"}
Sep 16 12:22:14.530885 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:14.530743    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88" event={"ID":"84958338-ef8c-42cf-ac40-5e37fce76d38","Type":"ContainerStarted","Data":"ebbc23fafb9abd0a5769b08c9e668639b08bc8162359da94cdb5fb328c3bff44"}
Sep 16 12:22:14.530885 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:14.530797    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"
Sep 16 12:22:14.532445 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:14.532419    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw" event={"ID":"bd0f5528-e83a-47c0-afde-db26417e5f2a","Type":"ContainerStarted","Data":"7bcf6e3073e8b56165222ecada583a3c521f85936d77812e834dd5e982f7ddcc"}
Sep 16 12:22:14.532593 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:14.532457    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw" event={"ID":"bd0f5528-e83a-47c0-afde-db26417e5f2a","Type":"ContainerStarted","Data":"ce529114f7544e142a1a32b935ab2a5407240c4ba029d6694d7cdd587ffda330"}
Sep 16 12:22:14.532593 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:14.532492    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"
Sep 16 12:22:14.546849 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:14.546805    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88" podStartSLOduration=1.5467884729999999 podStartE2EDuration="1.546788473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:22:14.544728487 +0000 UTC m=+1131.298687212" watchObservedRunningTime="2026-09-16 12:22:14.546788473 +0000 UTC m=+1131.300747205"
Sep 16 12:22:14.560290 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:14.560247    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw" podStartSLOduration=1.560233249 podStartE2EDuration="1.560233249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:22:14.55847855 +0000 UTC m=+1131.312437282" watchObservedRunningTime="2026-09-16 12:22:14.560233249 +0000 UTC m=+1131.314191980"
Sep 16 12:22:14.687884 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:14.687850    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:20.654231 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:20.654196    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:21.373683 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:21.373642    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:22.038898 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:22.038868    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:22.832227 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:22.832189    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:23.452730 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:23.452682    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:24.132897 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:24.132863    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:25.021908 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:25.021875    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:25.539414 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:25.539383    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"
Sep 16 12:22:25.539598 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:25.539451    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"
Sep 16 12:22:25.844614 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:25.844583    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:26.458091 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:26.458060    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:27.121171 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:27.121133    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:27.956936 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:27.956907    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:28.617159 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:28.617119    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:29.615988 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:29.615953    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:30.307997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:30.307962    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:31.052148 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:31.052108    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:31.694868 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:31.694835    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:32.320721 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:32.320683    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:32.931034 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:32.931000    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"]
Sep 16 12:22:32.931303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:32.931267    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm" podUID="244e6664-8f6f-4019-b8b7-bf86f29e474b" containerName="maas-api" containerID="cri-o://3eb550c42feb91187e39884036d65ed40beb24271eb55442ad4a2e518a0b711d" gracePeriod=30
Sep 16 12:22:32.960381 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:32.960358    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:33.053450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.052401    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:33.113743 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.113703    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:33.203976 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.203948    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:22:33.215643 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.215612    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:33.285395 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.285352    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-maas-api-tls\") pod \"244e6664-8f6f-4019-b8b7-bf86f29e474b\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") "
Sep 16 12:22:33.285395 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.285397    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244e6664-8f6f-4019-b8b7-bf86f29e474b-openshift-service-ca\") pod \"244e6664-8f6f-4019-b8b7-bf86f29e474b\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") "
Sep 16 12:22:33.285699 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.285449    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-metrics-certs\") pod \"244e6664-8f6f-4019-b8b7-bf86f29e474b\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") "
Sep 16 12:22:33.285699 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.285514    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5h5b\" (UniqueName: \"kubernetes.io/projected/244e6664-8f6f-4019-b8b7-bf86f29e474b-kube-api-access-m5h5b\") pod \"244e6664-8f6f-4019-b8b7-bf86f29e474b\" (UID: \"244e6664-8f6f-4019-b8b7-bf86f29e474b\") "
Sep 16 12:22:33.285861 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.285831    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244e6664-8f6f-4019-b8b7-bf86f29e474b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "244e6664-8f6f-4019-b8b7-bf86f29e474b" (UID: "244e6664-8f6f-4019-b8b7-bf86f29e474b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:33.288072 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.288046    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244e6664-8f6f-4019-b8b7-bf86f29e474b-kube-api-access-m5h5b" (OuterVolumeSpecName: "kube-api-access-m5h5b") pod "244e6664-8f6f-4019-b8b7-bf86f29e474b" (UID: "244e6664-8f6f-4019-b8b7-bf86f29e474b"). InnerVolumeSpecName "kube-api-access-m5h5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:33.288182 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.288105    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "244e6664-8f6f-4019-b8b7-bf86f29e474b" (UID: "244e6664-8f6f-4019-b8b7-bf86f29e474b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:22:33.288412 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.288386    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "244e6664-8f6f-4019-b8b7-bf86f29e474b" (UID: "244e6664-8f6f-4019-b8b7-bf86f29e474b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:22:33.291799 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.291777    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:33.386914 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.386877    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5h5b\" (UniqueName: \"kubernetes.io/projected/244e6664-8f6f-4019-b8b7-bf86f29e474b-kube-api-access-m5h5b\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:33.386914 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.386907    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-maas-api-tls\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:33.386914 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.386919    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244e6664-8f6f-4019-b8b7-bf86f29e474b-openshift-service-ca\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:33.387332 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.386928    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/244e6664-8f6f-4019-b8b7-bf86f29e474b-metrics-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:33.396539 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.396503    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:33.632990 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.632945    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:33.637498 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.637467    2558 generic.go:356] "Generic (PLEG): container finished" podID="244e6664-8f6f-4019-b8b7-bf86f29e474b" containerID="3eb550c42feb91187e39884036d65ed40beb24271eb55442ad4a2e518a0b711d" exitCode=0
Sep 16 12:22:33.637646 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.637513    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"
Sep 16 12:22:33.637646 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.637568    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm" event={"ID":"244e6664-8f6f-4019-b8b7-bf86f29e474b","Type":"ContainerDied","Data":"3eb550c42feb91187e39884036d65ed40beb24271eb55442ad4a2e518a0b711d"}
Sep 16 12:22:33.637721 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.637648    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm" event={"ID":"244e6664-8f6f-4019-b8b7-bf86f29e474b","Type":"ContainerDied","Data":"6fdc2e3b83fed6225f8abc8348c1b6e5661f0d2fe3c08b6986e91ed9bb0c3439"}
Sep 16 12:22:33.637721 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.637674    2558 scope.go:117] "RemoveContainer" containerID="3eb550c42feb91187e39884036d65ed40beb24271eb55442ad4a2e518a0b711d"
Sep 16 12:22:33.647483 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.647463    2558 scope.go:117] "RemoveContainer" containerID="3eb550c42feb91187e39884036d65ed40beb24271eb55442ad4a2e518a0b711d"
Sep 16 12:22:33.647801 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:22:33.647782    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb550c42feb91187e39884036d65ed40beb24271eb55442ad4a2e518a0b711d\": container with ID starting with 3eb550c42feb91187e39884036d65ed40beb24271eb55442ad4a2e518a0b711d not found: ID does not exist" containerID="3eb550c42feb91187e39884036d65ed40beb24271eb55442ad4a2e518a0b711d"
Sep 16 12:22:33.647855 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.647812    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb550c42feb91187e39884036d65ed40beb24271eb55442ad4a2e518a0b711d"} err="failed to get container status \"3eb550c42feb91187e39884036d65ed40beb24271eb55442ad4a2e518a0b711d\": rpc error: code = NotFound desc = could not find container \"3eb550c42feb91187e39884036d65ed40beb24271eb55442ad4a2e518a0b711d\": container with ID starting with 3eb550c42feb91187e39884036d65ed40beb24271eb55442ad4a2e518a0b711d not found: ID does not exist"
Sep 16 12:22:33.658894 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.658866    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"]
Sep 16 12:22:33.664132 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.664112    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-00a141-9bdf57677-9rxcm"]
Sep 16 12:22:33.758925 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:33.758897    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="244e6664-8f6f-4019-b8b7-bf86f29e474b" path="/var/lib/kubelet/pods/244e6664-8f6f-4019-b8b7-bf86f29e474b/volumes"
Sep 16 12:22:34.203977 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:34.203948    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:34.883589 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:34.883560    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:35.705453 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:35.705415    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:36.321717 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:36.321686    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:36.941216 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:36.941185    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:37.664037 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:37.664006    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:38.612460 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:38.612421    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"]
Sep 16 12:22:38.708847 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:38.708812    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:38.809612 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:38.809577    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:39.219711 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.219682    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:39.298107 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.298077    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"]
Sep 16 12:22:39.298374 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.298344    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw" podUID="bd0f5528-e83a-47c0-afde-db26417e5f2a" containerName="payload-processing" containerID="cri-o://7bcf6e3073e8b56165222ecada583a3c521f85936d77812e834dd5e982f7ddcc" gracePeriod=30
Sep 16 12:22:39.309962 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.309864    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"]
Sep 16 12:22:39.310120 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.310095    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88" podUID="84958338-ef8c-42cf-ac40-5e37fce76d38" containerName="payload-pre-processing" containerID="cri-o://8edf28bf9ba1526ee2d3b3d43918e1e31a598f8a747425cb93f2cd99e65f70bb" gracePeriod=30
Sep 16 12:22:39.333922 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.333893    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:39.433456 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.433423    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:39.534941 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.534910    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:39.586029 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.586002    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"
Sep 16 12:22:39.589245 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.589188    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"
Sep 16 12:22:39.621250 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.621225    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:39.647108 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.647076    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plr9d\" (UniqueName: \"kubernetes.io/projected/84958338-ef8c-42cf-ac40-5e37fce76d38-kube-api-access-plr9d\") pod \"84958338-ef8c-42cf-ac40-5e37fce76d38\" (UID: \"84958338-ef8c-42cf-ac40-5e37fce76d38\") "
Sep 16 12:22:39.647108 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.647116    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tjwf\" (UniqueName: \"kubernetes.io/projected/bd0f5528-e83a-47c0-afde-db26417e5f2a-kube-api-access-5tjwf\") pod \"bd0f5528-e83a-47c0-afde-db26417e5f2a\" (UID: \"bd0f5528-e83a-47c0-afde-db26417e5f2a\") "
Sep 16 12:22:39.647355 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.647151    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0f5528-e83a-47c0-afde-db26417e5f2a-plugins-config-volume\") pod \"bd0f5528-e83a-47c0-afde-db26417e5f2a\" (UID: \"bd0f5528-e83a-47c0-afde-db26417e5f2a\") "
Sep 16 12:22:39.647355 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.647260    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84958338-ef8c-42cf-ac40-5e37fce76d38-plugins-config-volume\") pod \"84958338-ef8c-42cf-ac40-5e37fce76d38\" (UID: \"84958338-ef8c-42cf-ac40-5e37fce76d38\") "
Sep 16 12:22:39.647658 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.647625    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd0f5528-e83a-47c0-afde-db26417e5f2a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd0f5528-e83a-47c0-afde-db26417e5f2a" (UID: "bd0f5528-e83a-47c0-afde-db26417e5f2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:39.647775 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.647641    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84958338-ef8c-42cf-ac40-5e37fce76d38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84958338-ef8c-42cf-ac40-5e37fce76d38" (UID: "84958338-ef8c-42cf-ac40-5e37fce76d38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:39.649929 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.649906    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84958338-ef8c-42cf-ac40-5e37fce76d38-kube-api-access-plr9d" (OuterVolumeSpecName: "kube-api-access-plr9d") pod "84958338-ef8c-42cf-ac40-5e37fce76d38" (UID: "84958338-ef8c-42cf-ac40-5e37fce76d38"). InnerVolumeSpecName "kube-api-access-plr9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:39.650042 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.649935    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0f5528-e83a-47c0-afde-db26417e5f2a-kube-api-access-5tjwf" (OuterVolumeSpecName: "kube-api-access-5tjwf") pod "bd0f5528-e83a-47c0-afde-db26417e5f2a" (UID: "bd0f5528-e83a-47c0-afde-db26417e5f2a"). InnerVolumeSpecName "kube-api-access-5tjwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:39.673060 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.673032    2558 generic.go:356] "Generic (PLEG): container finished" podID="84958338-ef8c-42cf-ac40-5e37fce76d38" containerID="8edf28bf9ba1526ee2d3b3d43918e1e31a598f8a747425cb93f2cd99e65f70bb" exitCode=0
Sep 16 12:22:39.673214 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.673075    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"
Sep 16 12:22:39.673214 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.673103    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88" event={"ID":"84958338-ef8c-42cf-ac40-5e37fce76d38","Type":"ContainerDied","Data":"8edf28bf9ba1526ee2d3b3d43918e1e31a598f8a747425cb93f2cd99e65f70bb"}
Sep 16 12:22:39.673214 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.673153    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88" event={"ID":"84958338-ef8c-42cf-ac40-5e37fce76d38","Type":"ContainerDied","Data":"ebbc23fafb9abd0a5769b08c9e668639b08bc8162359da94cdb5fb328c3bff44"}
Sep 16 12:22:39.673214 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.673178    2558 scope.go:117] "RemoveContainer" containerID="8edf28bf9ba1526ee2d3b3d43918e1e31a598f8a747425cb93f2cd99e65f70bb"
Sep 16 12:22:39.674530 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.674504    2558 generic.go:356] "Generic (PLEG): container finished" podID="bd0f5528-e83a-47c0-afde-db26417e5f2a" containerID="7bcf6e3073e8b56165222ecada583a3c521f85936d77812e834dd5e982f7ddcc" exitCode=0
Sep 16 12:22:39.674676 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.674571    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"
Sep 16 12:22:39.674676 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.674584    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw" event={"ID":"bd0f5528-e83a-47c0-afde-db26417e5f2a","Type":"ContainerDied","Data":"7bcf6e3073e8b56165222ecada583a3c521f85936d77812e834dd5e982f7ddcc"}
Sep 16 12:22:39.674676 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.674634    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw" event={"ID":"bd0f5528-e83a-47c0-afde-db26417e5f2a","Type":"ContainerDied","Data":"ce529114f7544e142a1a32b935ab2a5407240c4ba029d6694d7cdd587ffda330"}
Sep 16 12:22:39.688801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.687663    2558 scope.go:117] "RemoveContainer" containerID="8edf28bf9ba1526ee2d3b3d43918e1e31a598f8a747425cb93f2cd99e65f70bb"
Sep 16 12:22:39.688801 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:22:39.688073    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8edf28bf9ba1526ee2d3b3d43918e1e31a598f8a747425cb93f2cd99e65f70bb\": container with ID starting with 8edf28bf9ba1526ee2d3b3d43918e1e31a598f8a747425cb93f2cd99e65f70bb not found: ID does not exist" containerID="8edf28bf9ba1526ee2d3b3d43918e1e31a598f8a747425cb93f2cd99e65f70bb"
Sep 16 12:22:39.688801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.688110    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8edf28bf9ba1526ee2d3b3d43918e1e31a598f8a747425cb93f2cd99e65f70bb"} err="failed to get container status \"8edf28bf9ba1526ee2d3b3d43918e1e31a598f8a747425cb93f2cd99e65f70bb\": rpc error: code = NotFound desc = could not find container \"8edf28bf9ba1526ee2d3b3d43918e1e31a598f8a747425cb93f2cd99e65f70bb\": container with ID starting with 8edf28bf9ba1526ee2d3b3d43918e1e31a598f8a747425cb93f2cd99e65f70bb not found: ID does not exist"
Sep 16 12:22:39.688801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.688134    2558 scope.go:117] "RemoveContainer" containerID="7bcf6e3073e8b56165222ecada583a3c521f85936d77812e834dd5e982f7ddcc"
Sep 16 12:22:39.699785 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.699729    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"]
Sep 16 12:22:39.701731 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.701707    2558 scope.go:117] "RemoveContainer" containerID="7bcf6e3073e8b56165222ecada583a3c521f85936d77812e834dd5e982f7ddcc"
Sep 16 12:22:39.702060 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:22:39.702038    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bcf6e3073e8b56165222ecada583a3c521f85936d77812e834dd5e982f7ddcc\": container with ID starting with 7bcf6e3073e8b56165222ecada583a3c521f85936d77812e834dd5e982f7ddcc not found: ID does not exist" containerID="7bcf6e3073e8b56165222ecada583a3c521f85936d77812e834dd5e982f7ddcc"
Sep 16 12:22:39.702126 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.702067    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bcf6e3073e8b56165222ecada583a3c521f85936d77812e834dd5e982f7ddcc"} err="failed to get container status \"7bcf6e3073e8b56165222ecada583a3c521f85936d77812e834dd5e982f7ddcc\": rpc error: code = NotFound desc = could not find container \"7bcf6e3073e8b56165222ecada583a3c521f85936d77812e834dd5e982f7ddcc\": container with ID starting with 7bcf6e3073e8b56165222ecada583a3c521f85936d77812e834dd5e982f7ddcc not found: ID does not exist"
Sep 16 12:22:39.703185 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.703166    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2aae110-9d5d48658-c2s88"]
Sep 16 12:22:39.713981 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.713956    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"]
Sep 16 12:22:39.717809 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.717787    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2aae110-67d84bcf9c-f7cjw"]
Sep 16 12:22:39.748968 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.748867    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plr9d\" (UniqueName: \"kubernetes.io/projected/84958338-ef8c-42cf-ac40-5e37fce76d38-kube-api-access-plr9d\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:39.748968 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.748902    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tjwf\" (UniqueName: \"kubernetes.io/projected/bd0f5528-e83a-47c0-afde-db26417e5f2a-kube-api-access-5tjwf\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:39.748968 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.748919    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0f5528-e83a-47c0-afde-db26417e5f2a-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:39.748968 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.748932    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84958338-ef8c-42cf-ac40-5e37fce76d38-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:39.759906 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.759878    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84958338-ef8c-42cf-ac40-5e37fce76d38" path="/var/lib/kubelet/pods/84958338-ef8c-42cf-ac40-5e37fce76d38/volumes"
Sep 16 12:22:39.760260 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.760247    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd0f5528-e83a-47c0-afde-db26417e5f2a" path="/var/lib/kubelet/pods/bd0f5528-e83a-47c0-afde-db26417e5f2a/volumes"
Sep 16 12:22:39.938080 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:39.938037    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:40.293398 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:40.293302    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:40.388932 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:40.388898    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:40.490789 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:40.490757    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:40.702480 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:40.702443    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:41.817902 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:41.817871    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:42.111582 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:42.111538    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:42.863960 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:42.863926    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:43.522072 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:43.522039    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:47.673674 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:47.673584    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:47.728988 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:47.728946    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:47.794864 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:47.794835    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:47.894087 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:47.894053    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:48.006070 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:48.005958    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:48.395711 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:48.395675    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:53.631867 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.631822    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm" podUID="a2a94984-baa6-488b-b3ef-55a375dd6a67" containerName="main" containerID="cri-o://b561dd724482dfa979b2073dcb58f354f6ef3ac1c6512c6fb92d8200ea9d191e" gracePeriod=45
Sep 16 12:22:53.749417 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.749391    2558 generic.go:356] "Generic (PLEG): container finished" podID="a2a94984-baa6-488b-b3ef-55a375dd6a67" containerID="b561dd724482dfa979b2073dcb58f354f6ef3ac1c6512c6fb92d8200ea9d191e" exitCode=0
Sep 16 12:22:53.749518 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.749460    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm" event={"ID":"a2a94984-baa6-488b-b3ef-55a375dd6a67","Type":"ContainerDied","Data":"b561dd724482dfa979b2073dcb58f354f6ef3ac1c6512c6fb92d8200ea9d191e"}
Sep 16 12:22:53.886582 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.886500    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:22:53.988263 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.988227    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmwwv\" (UniqueName: \"kubernetes.io/projected/a2a94984-baa6-488b-b3ef-55a375dd6a67-kube-api-access-cmwwv\") pod \"a2a94984-baa6-488b-b3ef-55a375dd6a67\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") "
Sep 16 12:22:53.988455 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.988289    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a94984-baa6-488b-b3ef-55a375dd6a67-tls-certs\") pod \"a2a94984-baa6-488b-b3ef-55a375dd6a67\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") "
Sep 16 12:22:53.988455 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.988361    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-dshm\") pod \"a2a94984-baa6-488b-b3ef-55a375dd6a67\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") "
Sep 16 12:22:53.988455 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.988386    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-model-cache\") pod \"a2a94984-baa6-488b-b3ef-55a375dd6a67\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") "
Sep 16 12:22:53.988658 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.988490    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-tmp-dir\") pod \"a2a94984-baa6-488b-b3ef-55a375dd6a67\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") "
Sep 16 12:22:53.988658 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.988533    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-home\") pod \"a2a94984-baa6-488b-b3ef-55a375dd6a67\" (UID: \"a2a94984-baa6-488b-b3ef-55a375dd6a67\") "
Sep 16 12:22:53.988658 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.988623    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2a94984-baa6-488b-b3ef-55a375dd6a67" (UID: "a2a94984-baa6-488b-b3ef-55a375dd6a67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:53.988833 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.988747    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2a94984-baa6-488b-b3ef-55a375dd6a67" (UID: "a2a94984-baa6-488b-b3ef-55a375dd6a67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:53.988884 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.988855    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-home" (OuterVolumeSpecName: "home") pod "a2a94984-baa6-488b-b3ef-55a375dd6a67" (UID: "a2a94984-baa6-488b-b3ef-55a375dd6a67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:53.988962 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.988942    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-model-cache\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:53.989016 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.988962    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-tmp-dir\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:53.989016 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.988971    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-home\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:53.990582 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.990541    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a94984-baa6-488b-b3ef-55a375dd6a67-kube-api-access-cmwwv" (OuterVolumeSpecName: "kube-api-access-cmwwv") pod "a2a94984-baa6-488b-b3ef-55a375dd6a67" (UID: "a2a94984-baa6-488b-b3ef-55a375dd6a67"). InnerVolumeSpecName "kube-api-access-cmwwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:53.990852 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.990833    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2a94984-baa6-488b-b3ef-55a375dd6a67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2a94984-baa6-488b-b3ef-55a375dd6a67" (UID: "a2a94984-baa6-488b-b3ef-55a375dd6a67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:22:53.990852 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:53.990830    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-dshm" (OuterVolumeSpecName: "dshm") pod "a2a94984-baa6-488b-b3ef-55a375dd6a67" (UID: "a2a94984-baa6-488b-b3ef-55a375dd6a67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:54.090019 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:54.089977    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a94984-baa6-488b-b3ef-55a375dd6a67-dshm\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:54.090019 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:54.090013    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmwwv\" (UniqueName: \"kubernetes.io/projected/a2a94984-baa6-488b-b3ef-55a375dd6a67-kube-api-access-cmwwv\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:54.090019 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:54.090024    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a94984-baa6-488b-b3ef-55a375dd6a67-tls-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:22:54.755764 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:54.755736    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"
Sep 16 12:22:54.755764 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:54.755750    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm" event={"ID":"a2a94984-baa6-488b-b3ef-55a375dd6a67","Type":"ContainerDied","Data":"5a9cac5cd5aeac9a5eaa30acdee8d26ab351416113961ad75bb4ddf43df6f256"}
Sep 16 12:22:54.756237 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:54.755803    2558 scope.go:117] "RemoveContainer" containerID="b561dd724482dfa979b2073dcb58f354f6ef3ac1c6512c6fb92d8200ea9d191e"
Sep 16 12:22:54.775753 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:54.775729    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"]
Sep 16 12:22:54.781574 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:54.781535    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-00a141/autores-model-00a141-kserve-cc86989d7-4kbwm"]
Sep 16 12:22:55.763350 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:55.763317    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2a94984-baa6-488b-b3ef-55a375dd6a67" path="/var/lib/kubelet/pods/a2a94984-baa6-488b-b3ef-55a375dd6a67/volumes"
Sep 16 12:22:59.237526 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:22:59.237493    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:15.429828 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:15.429795    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:19.305860 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:19.305824    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:20.665357 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:20.665327    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:21.092757 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:21.092724    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:21.382716 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:21.382617    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:21.482808 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:21.482775    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:24.005606 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:24.005480    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:24.814839 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:24.814807    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:24.915298 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:24.915254    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:25.623315 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:25.623279    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:25.857036 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:25.856991    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:25.959495 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:25.959400    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:26.057614 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:26.057580    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:26.316509 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:26.316383    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:27.035925 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:27.035876    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:27.854778 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:27.854744    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:27.956698 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:27.956662    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:28.302511 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.302423    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:28.434344 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.434316    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:28.536375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.536346    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:28.543964 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.543939    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"]
Sep 16 12:23:28.544315 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.544303    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="244e6664-8f6f-4019-b8b7-bf86f29e474b" containerName="maas-api"
Sep 16 12:23:28.544358 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.544317    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="244e6664-8f6f-4019-b8b7-bf86f29e474b" containerName="maas-api"
Sep 16 12:23:28.544358 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.544326    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84958338-ef8c-42cf-ac40-5e37fce76d38" containerName="payload-pre-processing"
Sep 16 12:23:28.544358 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.544332    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="84958338-ef8c-42cf-ac40-5e37fce76d38" containerName="payload-pre-processing"
Sep 16 12:23:28.544358 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.544343    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd0f5528-e83a-47c0-afde-db26417e5f2a" containerName="payload-processing"
Sep 16 12:23:28.544358 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.544349    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd0f5528-e83a-47c0-afde-db26417e5f2a" containerName="payload-processing"
Sep 16 12:23:28.544511 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.544361    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2a94984-baa6-488b-b3ef-55a375dd6a67" containerName="main"
Sep 16 12:23:28.544511 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.544367    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2a94984-baa6-488b-b3ef-55a375dd6a67" containerName="main"
Sep 16 12:23:28.544511 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.544425    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2a94984-baa6-488b-b3ef-55a375dd6a67" containerName="main"
Sep 16 12:23:28.544511 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.544432    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd0f5528-e83a-47c0-afde-db26417e5f2a" containerName="payload-processing"
Sep 16 12:23:28.544511 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.544437    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="84958338-ef8c-42cf-ac40-5e37fce76d38" containerName="payload-pre-processing"
Sep 16 12:23:28.544511 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.544444    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="244e6664-8f6f-4019-b8b7-bf86f29e474b" containerName="maas-api"
Sep 16 12:23:28.548877 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.548860    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"
Sep 16 12:23:28.550818 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.550798    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f1730c-dockercfg-vlr9p\""
Sep 16 12:23:28.550924 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.550803    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-f1730c\""
Sep 16 12:23:28.558197 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.558175    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"]
Sep 16 12:23:28.578393 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.578373    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"]
Sep 16 12:23:28.581803 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.581786    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"
Sep 16 12:23:28.593458 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.593438    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"]
Sep 16 12:23:28.709771 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.709741    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71ce862-10f8-45f6-919d-17de2c394379-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z\" (UID: \"a71ce862-10f8-45f6-919d-17de2c394379\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"
Sep 16 12:23:28.709934 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.709791    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf4bz\" (UniqueName: \"kubernetes.io/projected/a71ce862-10f8-45f6-919d-17de2c394379-kube-api-access-cf4bz\") pod \"payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z\" (UID: \"a71ce862-10f8-45f6-919d-17de2c394379\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"
Sep 16 12:23:28.709934 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.709824    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf55w\" (UniqueName: \"kubernetes.io/projected/d3145f96-225a-406c-b78a-411cec78fef8-kube-api-access-wf55w\") pod \"payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p\" (UID: \"d3145f96-225a-406c-b78a-411cec78fef8\") " pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"
Sep 16 12:23:28.710054 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.710009    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3145f96-225a-406c-b78a-411cec78fef8-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p\" (UID: \"d3145f96-225a-406c-b78a-411cec78fef8\") " pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"
Sep 16 12:23:28.810788 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.810700    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf55w\" (UniqueName: \"kubernetes.io/projected/d3145f96-225a-406c-b78a-411cec78fef8-kube-api-access-wf55w\") pod \"payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p\" (UID: \"d3145f96-225a-406c-b78a-411cec78fef8\") " pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"
Sep 16 12:23:28.810788 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.810776    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3145f96-225a-406c-b78a-411cec78fef8-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p\" (UID: \"d3145f96-225a-406c-b78a-411cec78fef8\") " pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"
Sep 16 12:23:28.810996 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.810844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71ce862-10f8-45f6-919d-17de2c394379-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z\" (UID: \"a71ce862-10f8-45f6-919d-17de2c394379\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"
Sep 16 12:23:28.810996 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.810892    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4bz\" (UniqueName: \"kubernetes.io/projected/a71ce862-10f8-45f6-919d-17de2c394379-kube-api-access-cf4bz\") pod \"payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z\" (UID: \"a71ce862-10f8-45f6-919d-17de2c394379\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"
Sep 16 12:23:28.811587 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.811540    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3145f96-225a-406c-b78a-411cec78fef8-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p\" (UID: \"d3145f96-225a-406c-b78a-411cec78fef8\") " pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"
Sep 16 12:23:28.811704 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.811648    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71ce862-10f8-45f6-919d-17de2c394379-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z\" (UID: \"a71ce862-10f8-45f6-919d-17de2c394379\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"
Sep 16 12:23:28.820268 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.820248    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf55w\" (UniqueName: \"kubernetes.io/projected/d3145f96-225a-406c-b78a-411cec78fef8-kube-api-access-wf55w\") pod \"payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p\" (UID: \"d3145f96-225a-406c-b78a-411cec78fef8\") " pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"
Sep 16 12:23:28.820373 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.820352    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf4bz\" (UniqueName: \"kubernetes.io/projected/a71ce862-10f8-45f6-919d-17de2c394379-kube-api-access-cf4bz\") pod \"payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z\" (UID: \"a71ce862-10f8-45f6-919d-17de2c394379\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"
Sep 16 12:23:28.861061 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.861039    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"
Sep 16 12:23:28.892177 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:28.892150    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"
Sep 16 12:23:29.135147 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:29.135124    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"]
Sep 16 12:23:29.137132 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:23:29.137106    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71ce862_10f8_45f6_919d_17de2c394379.slice/crio-9863b5ce581bf8c48a32c2daa8c0fdceae81cdb9cb86962e87a4e7ccd8d0aeee WatchSource:0}: Error finding container 9863b5ce581bf8c48a32c2daa8c0fdceae81cdb9cb86962e87a4e7ccd8d0aeee: Status 404 returned error can't find the container with id 9863b5ce581bf8c48a32c2daa8c0fdceae81cdb9cb86962e87a4e7ccd8d0aeee
Sep 16 12:23:29.171824 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:29.171803    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"]
Sep 16 12:23:29.174054 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:23:29.174031    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3145f96_225a_406c_b78a_411cec78fef8.slice/crio-5f67f8f28bc171c0987de39fe8389e5162db17421851334f7118fd15cc25380a WatchSource:0}: Error finding container 5f67f8f28bc171c0987de39fe8389e5162db17421851334f7118fd15cc25380a: Status 404 returned error can't find the container with id 5f67f8f28bc171c0987de39fe8389e5162db17421851334f7118fd15cc25380a
Sep 16 12:23:29.932716 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:29.932673    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z" event={"ID":"a71ce862-10f8-45f6-919d-17de2c394379","Type":"ContainerStarted","Data":"69b532d5b187eabf1f6b12efe26acdbc801ac81d892b97284eb02a8dc79dc315"}
Sep 16 12:23:29.933183 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:29.932745    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z" event={"ID":"a71ce862-10f8-45f6-919d-17de2c394379","Type":"ContainerStarted","Data":"9863b5ce581bf8c48a32c2daa8c0fdceae81cdb9cb86962e87a4e7ccd8d0aeee"}
Sep 16 12:23:29.933183 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:29.932777    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"
Sep 16 12:23:29.934240 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:29.934211    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p" event={"ID":"d3145f96-225a-406c-b78a-411cec78fef8","Type":"ContainerStarted","Data":"66be1904e6f831cd168148e3915ea4fa55c13a1233a930ef4d88fdab96f623c7"}
Sep 16 12:23:29.934378 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:29.934268    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"
Sep 16 12:23:29.934378 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:29.934291    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p" event={"ID":"d3145f96-225a-406c-b78a-411cec78fef8","Type":"ContainerStarted","Data":"5f67f8f28bc171c0987de39fe8389e5162db17421851334f7118fd15cc25380a"}
Sep 16 12:23:29.948211 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:29.948143    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z" podStartSLOduration=1.948125766 podStartE2EDuration="1.948125766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:23:29.947116323 +0000 UTC m=+1206.701075056" watchObservedRunningTime="2026-09-16 12:23:29.948125766 +0000 UTC m=+1206.702084502"
Sep 16 12:23:29.963144 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:29.963101    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p" podStartSLOduration=1.963084273 podStartE2EDuration="1.963084273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:23:29.962673535 +0000 UTC m=+1206.716632269" watchObservedRunningTime="2026-09-16 12:23:29.963084273 +0000 UTC m=+1206.717043007"
Sep 16 12:23:40.941076 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:40.941043    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"
Sep 16 12:23:40.941575 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:40.941169    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"
Sep 16 12:23:41.849902 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:41.849868    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:44.855320 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:44.855285    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:46.199468 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:46.199424    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:47.538806 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:47.538774    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:47.599998 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:47.599967    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:47.680903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:47.680871    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:47.780758 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:47.780720    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:49.730440 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:49.730389    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:49.947141 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:49.947103    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:50.618916 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:50.618884    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:51.434089 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:51.434057    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:52.177897 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:52.177841    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:52.828761 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:52.828723    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:53.554411 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:53.554374    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:58.210020 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:58.209988    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:58.965787 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:23:58.965756    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:04.719304 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:04.719267    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:04.824245 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:04.824201    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:04.922118 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:04.922085    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:05.020383 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:05.020294    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:05.652441 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:05.652412    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:05.753276 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:05.753239    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:05.849700 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:05.849674    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:07.260519 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:07.260485    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:08.462361 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:08.462329    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:12.993262 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:12.993226    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:13.092956 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:13.092926    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:13.630220 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:13.630183    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:19.328562 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.328469    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:19.397994 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.397954    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:19.554320 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.554270    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:19.832892 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.832858    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"]
Sep 16 12:24:19.836783 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.836759    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:19.839671 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.839650    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cadab1d8-openshift-default-dockercfg-grt6f\""
Sep 16 12:24:19.850374 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.850349    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"]
Sep 16 12:24:19.875013 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.874949    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:19.969994 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.969958    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:19.986729 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.986698    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-podinfo\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:19.986892 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.986756    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f71290-9ae1-4f1e-8b56-d13f69990f93-istiod-ca-cert\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:19.986892 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.986806    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-token\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:19.986892 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.986887    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-workload-socket\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:19.987043 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.986925    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-data\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:19.987043 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.986961    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-workload-certs\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:19.987127 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.987042    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpt2x\" (UniqueName: \"kubernetes.io/projected/65f71290-9ae1-4f1e-8b56-d13f69990f93-kube-api-access-hpt2x\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:19.987127 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.987078    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-envoy\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:19.987127 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:19.987106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-credential-socket\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.088031 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.087930    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-workload-socket\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.088031 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.087981    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-data\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.088031 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.088016    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-workload-certs\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.088314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.088068    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpt2x\" (UniqueName: \"kubernetes.io/projected/65f71290-9ae1-4f1e-8b56-d13f69990f93-kube-api-access-hpt2x\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.088314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.088096    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-envoy\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.088314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.088127    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-credential-socket\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.088314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.088158    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-podinfo\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.088314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.088200    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f71290-9ae1-4f1e-8b56-d13f69990f93-istiod-ca-cert\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.088314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.088228    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-token\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.088653 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.088384    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-workload-socket\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.088653 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.088601    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-workload-certs\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.088752 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.088691    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-data\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.088752 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.088724    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-credential-socket\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.089095 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.089076    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f71290-9ae1-4f1e-8b56-d13f69990f93-istiod-ca-cert\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.090598 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.090577    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-envoy\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.090838 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.090817    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-podinfo\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.095434 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.095407    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-token\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.095765 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.095745    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpt2x\" (UniqueName: \"kubernetes.io/projected/65f71290-9ae1-4f1e-8b56-d13f69990f93-kube-api-access-hpt2x\") pod \"e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") " pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.148710 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.148672    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:20.468394 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.468363    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"]
Sep 16 12:24:20.471773 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:24:20.471746    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f71290_9ae1_4f1e_8b56_d13f69990f93.slice/crio-5a717cf8f168433a392e87ae0fe88c1c55f36803bccf6e652aeea34a667f4be7 WatchSource:0}: Error finding container 5a717cf8f168433a392e87ae0fe88c1c55f36803bccf6e652aeea34a667f4be7: Status 404 returned error can't find the container with id 5a717cf8f168433a392e87ae0fe88c1c55f36803bccf6e652aeea34a667f4be7
Sep 16 12:24:20.477064 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.477023    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:24:20.477145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.477114    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:24:20.477192 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:20.477155    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:24:21.190173 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:21.190136    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm" event={"ID":"65f71290-9ae1-4f1e-8b56-d13f69990f93","Type":"ContainerStarted","Data":"e3caf819a1cafdd6c558c1d71697d04da085cdeac59a1918cdfae1a4bd6deb1d"}
Sep 16 12:24:21.190349 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:21.190219    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm" event={"ID":"65f71290-9ae1-4f1e-8b56-d13f69990f93","Type":"ContainerStarted","Data":"5a717cf8f168433a392e87ae0fe88c1c55f36803bccf6e652aeea34a667f4be7"}
Sep 16 12:24:21.229904 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:21.229845    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm" podStartSLOduration=2.2298292809999998 podStartE2EDuration="2.229829281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:24:21.226836294 +0000 UTC m=+1257.980795024" watchObservedRunningTime="2026-09-16 12:24:21.229829281 +0000 UTC m=+1257.983788012"
Sep 16 12:24:22.149462 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:22.149424    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:22.154459 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:22.154433    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:22.194980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:22.194958    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:22.195732 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:22.195713    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:25.640146 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:25.640098    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:25.904466 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:25.904380    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:26.147339 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:26.147308    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:26.247721 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:26.247606    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:26.440314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:26.440259    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:27.217792 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:27.217758    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:27.861761 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:27.861729    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:28.724636 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:28.724607    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:29.373906 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:29.373869    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:29.487951 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:29.487920    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:29.580396 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:29.580364    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:29.826664 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:29.826581    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:29.913217 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:29.913186    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:29.920054 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:29.920029    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"]
Sep 16 12:24:29.923731 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:29.923712    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"
Sep 16 12:24:29.926078 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:29.926048    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cadab1d8-dockercfg-w4ngh\""
Sep 16 12:24:29.926186 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:29.926102    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cadab1d8\""
Sep 16 12:24:29.934924 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:29.934895    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"]
Sep 16 12:24:29.983676 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:29.983643    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59216a8c-9e0b-4e43-bf95-c3f82e5610c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8\" (UID: \"59216a8c-9e0b-4e43-bf95-c3f82e5610c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"
Sep 16 12:24:29.983823 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:29.983756    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2cg8\" (UniqueName: \"kubernetes.io/projected/59216a8c-9e0b-4e43-bf95-c3f82e5610c6-kube-api-access-f2cg8\") pod \"payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8\" (UID: \"59216a8c-9e0b-4e43-bf95-c3f82e5610c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"
Sep 16 12:24:30.084602 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:30.084485    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2cg8\" (UniqueName: \"kubernetes.io/projected/59216a8c-9e0b-4e43-bf95-c3f82e5610c6-kube-api-access-f2cg8\") pod \"payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8\" (UID: \"59216a8c-9e0b-4e43-bf95-c3f82e5610c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"
Sep 16 12:24:30.084777 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:30.084608    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59216a8c-9e0b-4e43-bf95-c3f82e5610c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8\" (UID: \"59216a8c-9e0b-4e43-bf95-c3f82e5610c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"
Sep 16 12:24:30.085692 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:30.085666    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59216a8c-9e0b-4e43-bf95-c3f82e5610c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8\" (UID: \"59216a8c-9e0b-4e43-bf95-c3f82e5610c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"
Sep 16 12:24:30.094124 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:30.094100    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2cg8\" (UniqueName: \"kubernetes.io/projected/59216a8c-9e0b-4e43-bf95-c3f82e5610c6-kube-api-access-f2cg8\") pod \"payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8\" (UID: \"59216a8c-9e0b-4e43-bf95-c3f82e5610c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"
Sep 16 12:24:30.234379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:30.234348    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"
Sep 16 12:24:30.508115 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:30.508082    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"]
Sep 16 12:24:30.509818 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:24:30.509780    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59216a8c_9e0b_4e43_bf95_c3f82e5610c6.slice/crio-0afb44601034c24f41873b610ae6d7b3da7c8781871af64ac71ddf911ae82fe8 WatchSource:0}: Error finding container 0afb44601034c24f41873b610ae6d7b3da7c8781871af64ac71ddf911ae82fe8: Status 404 returned error can't find the container with id 0afb44601034c24f41873b610ae6d7b3da7c8781871af64ac71ddf911ae82fe8
Sep 16 12:24:31.244346 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:31.244308    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8" event={"ID":"59216a8c-9e0b-4e43-bf95-c3f82e5610c6","Type":"ContainerStarted","Data":"6cffb8dea70ba9ae3feea1c3333692f04d113605f9bc1ddb6ae9896e4527e2f2"}
Sep 16 12:24:31.244767 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:31.244398    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8" event={"ID":"59216a8c-9e0b-4e43-bf95-c3f82e5610c6","Type":"ContainerStarted","Data":"0afb44601034c24f41873b610ae6d7b3da7c8781871af64ac71ddf911ae82fe8"}
Sep 16 12:24:31.244767 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:31.244421    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"
Sep 16 12:24:31.258804 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:31.258751    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8" podStartSLOduration=2.258732006 podStartE2EDuration="2.258732006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:24:31.258225115 +0000 UTC m=+1268.012183870" watchObservedRunningTime="2026-09-16 12:24:31.258732006 +0000 UTC m=+1268.012690737"
Sep 16 12:24:41.266838 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:41.266794    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:42.252808 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:42.252764    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"
Sep 16 12:24:50.345390 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:50.345355    2558 generic.go:356] "Generic (PLEG): container finished" podID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerID="e3caf819a1cafdd6c558c1d71697d04da085cdeac59a1918cdfae1a4bd6deb1d" exitCode=0
Sep 16 12:24:50.345818 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:50.345430    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm" event={"ID":"65f71290-9ae1-4f1e-8b56-d13f69990f93","Type":"ContainerDied","Data":"e3caf819a1cafdd6c558c1d71697d04da085cdeac59a1918cdfae1a4bd6deb1d"}
Sep 16 12:24:50.345873 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:50.345857    2558 scope.go:117] "RemoveContainer" containerID="e3caf819a1cafdd6c558c1d71697d04da085cdeac59a1918cdfae1a4bd6deb1d"
Sep 16 12:24:50.347178 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:50.347150    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:24:50.347234 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:50.347220    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:24:50.347270 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:50.347254    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:24:51.356922 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:51.356881    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm" event={"ID":"65f71290-9ae1-4f1e-8b56-d13f69990f93","Type":"ContainerStarted","Data":"0302e6568c954239e5a0501c7ccbb0f816289fcfeb13cd1f871e72be2ea9b72f"}
Sep 16 12:24:52.150566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:52.150510    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:52.150758 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:52.150704    2558 patch_prober.go:28] interesting pod/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 12:24:52.150827 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:52.150759    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm" podUID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 12:24:52.195662 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:52.195632    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:52.320072 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:52.320034    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:53.149810 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:53.149763    2558 patch_prober.go:28] interesting pod/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 12:24:53.150289 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:53.149853    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm" podUID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 12:24:53.515727 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:53.515639    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:53.914133 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:53.914104    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:54.149285 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:54.149227    2558 patch_prober.go:28] interesting pod/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 12:24:54.149453 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:54.149319    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm" podUID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 12:24:54.628951 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:54.628922    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:55.149254 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:55.149207    2558 patch_prober.go:28] interesting pod/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 12:24:55.149458 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:55.149278    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm" podUID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 12:24:55.509302 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:55.509192    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:55.930910 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:55.930871    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:56.149915 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:56.149872    2558 patch_prober.go:28] interesting pod/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 12:24:56.150120 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:56.149956    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm" podUID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 12:24:56.160155 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:56.160121    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:56.571568 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:56.571516    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:56.671873 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:56.671832    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:57.152966 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:57.152938    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:57.153857 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:57.153834    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:24:57.195995 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:57.195967    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:57.291443 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:24:57.291409    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:02.535538 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.535499    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"]
Sep 16 12:25:02.539347 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.539325    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.541606 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.541584    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f1730c\"/\"openshift-service-ca.crt\""
Sep 16 12:25:02.543704 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.543672    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f1730c\"/\"kube-root-ca.crt\""
Sep 16 12:25:02.543805 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.543702    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f1730c\"/\"default-dockercfg-gjwcf\""
Sep 16 12:25:02.544006 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.543986    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f1730c\"/\"explicit-model-f1730c-kserve-self-signed-certs\""
Sep 16 12:25:02.551388 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.551359    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"]
Sep 16 12:25:02.561556 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.561492    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:02.696496 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.696461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xptzd\" (UniqueName: \"kubernetes.io/projected/b6614e64-c2c2-4985-a08a-52e37b88da92-kube-api-access-xptzd\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.696680 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.696523    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-home\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.696680 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.696593    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-model-cache\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.696680 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.696627    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-tmp-dir\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.696680 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.696668    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-dshm\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.696835 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.696712    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6614e64-c2c2-4985-a08a-52e37b88da92-tls-certs\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.717124 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.717095    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:02.797975 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.797896    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xptzd\" (UniqueName: \"kubernetes.io/projected/b6614e64-c2c2-4985-a08a-52e37b88da92-kube-api-access-xptzd\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.797975 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.797960    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-home\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.798198 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.797993    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-model-cache\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.798198 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.798011    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-tmp-dir\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.798198 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.798044    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-dshm\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.798198 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.798088    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6614e64-c2c2-4985-a08a-52e37b88da92-tls-certs\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.798450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.798423    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-home\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.798512 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.798437    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-tmp-dir\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.798512 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.798472    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-model-cache\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.800368 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.800348    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-dshm\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.800562 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.800527    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6614e64-c2c2-4985-a08a-52e37b88da92-tls-certs\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.805270 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.805249    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xptzd\" (UniqueName: \"kubernetes.io/projected/b6614e64-c2c2-4985-a08a-52e37b88da92-kube-api-access-xptzd\") pod \"explicit-model-f1730c-kserve-7688dd85b9-zd9cs\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") " pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:02.852991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:02.852966    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:03.056407 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:03.056379    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:03.125447 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:03.125416    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"]
Sep 16 12:25:03.127254 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:25:03.127229    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6614e64_c2c2_4985_a08a_52e37b88da92.slice/crio-5bfa64c633f79342d86397acbc582eaa8b9ecfb3964cb20e378b528396a6ebc1 WatchSource:0}: Error finding container 5bfa64c633f79342d86397acbc582eaa8b9ecfb3964cb20e378b528396a6ebc1: Status 404 returned error can't find the container with id 5bfa64c633f79342d86397acbc582eaa8b9ecfb3964cb20e378b528396a6ebc1
Sep 16 12:25:03.427469 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:03.427436    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs" event={"ID":"b6614e64-c2c2-4985-a08a-52e37b88da92","Type":"ContainerStarted","Data":"63fe38f5a48592767367c05b746186dce5cba71017412dc11c96a406475436b5"}
Sep 16 12:25:03.427669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:03.427479    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs" event={"ID":"b6614e64-c2c2-4985-a08a-52e37b88da92","Type":"ContainerStarted","Data":"5bfa64c633f79342d86397acbc582eaa8b9ecfb3964cb20e378b528396a6ebc1"}
Sep 16 12:25:03.445404 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:03.445356    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs" podStartSLOduration=1.445338273 podStartE2EDuration="1.445338273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:25:03.443275367 +0000 UTC m=+1300.197234110" watchObservedRunningTime="2026-09-16 12:25:03.445338273 +0000 UTC m=+1300.199297034"
Sep 16 12:25:03.716148 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:03.716063    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:04.336571 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:04.336518    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:05.129530 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:05.129499    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:06.094733 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:06.094700    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:06.546348 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:06.546257    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:07.239747 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:07.239715    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:08.018891 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:08.018852    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:12.520738 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:12.520700    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"]
Sep 16 12:25:12.539872 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:12.539840    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:12.753865 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:12.753829    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:12.854136 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:12.854103    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:13.431557 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.431506    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:13.457091 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.457056    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"]
Sep 16 12:25:13.457391 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.457363    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8" podUID="59216a8c-9e0b-4e43-bf95-c3f82e5610c6" containerName="payload-pre-processing" containerID="cri-o://6cffb8dea70ba9ae3feea1c3333692f04d113605f9bc1ddb6ae9896e4527e2f2" gracePeriod=30
Sep 16 12:25:13.528640 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.528613    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:13.599136 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.599101    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:13.656003 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.655969    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:13.717224 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.716882    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"
Sep 16 12:25:13.799860 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.799823    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59216a8c-9e0b-4e43-bf95-c3f82e5610c6-plugins-config-volume\") pod \"59216a8c-9e0b-4e43-bf95-c3f82e5610c6\" (UID: \"59216a8c-9e0b-4e43-bf95-c3f82e5610c6\") "
Sep 16 12:25:13.800022 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.799883    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2cg8\" (UniqueName: \"kubernetes.io/projected/59216a8c-9e0b-4e43-bf95-c3f82e5610c6-kube-api-access-f2cg8\") pod \"59216a8c-9e0b-4e43-bf95-c3f82e5610c6\" (UID: \"59216a8c-9e0b-4e43-bf95-c3f82e5610c6\") "
Sep 16 12:25:13.800201 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.800179    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59216a8c-9e0b-4e43-bf95-c3f82e5610c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59216a8c-9e0b-4e43-bf95-c3f82e5610c6" (UID: "59216a8c-9e0b-4e43-bf95-c3f82e5610c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:25:13.802074 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.802051    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59216a8c-9e0b-4e43-bf95-c3f82e5610c6-kube-api-access-f2cg8" (OuterVolumeSpecName: "kube-api-access-f2cg8") pod "59216a8c-9e0b-4e43-bf95-c3f82e5610c6" (UID: "59216a8c-9e0b-4e43-bf95-c3f82e5610c6"). InnerVolumeSpecName "kube-api-access-f2cg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:13.803840 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.803818    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:13.901009 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.900975    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59216a8c-9e0b-4e43-bf95-c3f82e5610c6-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:13.901009 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.901005    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2cg8\" (UniqueName: \"kubernetes.io/projected/59216a8c-9e0b-4e43-bf95-c3f82e5610c6-kube-api-access-f2cg8\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:13.962183 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:13.962095    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:14.497111 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:14.497074    2558 generic.go:356] "Generic (PLEG): container finished" podID="59216a8c-9e0b-4e43-bf95-c3f82e5610c6" containerID="6cffb8dea70ba9ae3feea1c3333692f04d113605f9bc1ddb6ae9896e4527e2f2" exitCode=0
Sep 16 12:25:14.497297 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:14.497122    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"
Sep 16 12:25:14.497297 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:14.497163    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8" event={"ID":"59216a8c-9e0b-4e43-bf95-c3f82e5610c6","Type":"ContainerDied","Data":"6cffb8dea70ba9ae3feea1c3333692f04d113605f9bc1ddb6ae9896e4527e2f2"}
Sep 16 12:25:14.497297 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:14.497213    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8" event={"ID":"59216a8c-9e0b-4e43-bf95-c3f82e5610c6","Type":"ContainerDied","Data":"0afb44601034c24f41873b610ae6d7b3da7c8781871af64ac71ddf911ae82fe8"}
Sep 16 12:25:14.497297 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:14.497239    2558 scope.go:117] "RemoveContainer" containerID="6cffb8dea70ba9ae3feea1c3333692f04d113605f9bc1ddb6ae9896e4527e2f2"
Sep 16 12:25:14.506408 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:14.506392    2558 scope.go:117] "RemoveContainer" containerID="6cffb8dea70ba9ae3feea1c3333692f04d113605f9bc1ddb6ae9896e4527e2f2"
Sep 16 12:25:14.506701 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:25:14.506679    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cffb8dea70ba9ae3feea1c3333692f04d113605f9bc1ddb6ae9896e4527e2f2\": container with ID starting with 6cffb8dea70ba9ae3feea1c3333692f04d113605f9bc1ddb6ae9896e4527e2f2 not found: ID does not exist" containerID="6cffb8dea70ba9ae3feea1c3333692f04d113605f9bc1ddb6ae9896e4527e2f2"
Sep 16 12:25:14.506785 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:14.506710    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cffb8dea70ba9ae3feea1c3333692f04d113605f9bc1ddb6ae9896e4527e2f2"} err="failed to get container status \"6cffb8dea70ba9ae3feea1c3333692f04d113605f9bc1ddb6ae9896e4527e2f2\": rpc error: code = NotFound desc = could not find container \"6cffb8dea70ba9ae3feea1c3333692f04d113605f9bc1ddb6ae9896e4527e2f2\": container with ID starting with 6cffb8dea70ba9ae3feea1c3333692f04d113605f9bc1ddb6ae9896e4527e2f2 not found: ID does not exist"
Sep 16 12:25:14.540988 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:14.540948    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:14.541334 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:14.541040    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"]
Sep 16 12:25:14.550079 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:14.550057    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cadab1d8-7865d97d4b-5s6l8"]
Sep 16 12:25:15.426978 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:15.426947    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:15.758698 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:15.758619    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59216a8c-9e0b-4e43-bf95-c3f82e5610c6" path="/var/lib/kubelet/pods/59216a8c-9e0b-4e43-bf95-c3f82e5610c6/volumes"
Sep 16 12:25:16.133800 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:16.133770    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:16.769205 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:16.769160    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:17.392640 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:17.392604    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:18.130918 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:18.130881    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:19.784243 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:19.784210    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:19.853060 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:19.853025    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:20.023646 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:20.023612    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"]
Sep 16 12:25:20.023862 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:20.023825    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm" podUID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerName="istio-proxy" containerID="cri-o://0302e6568c954239e5a0501c7ccbb0f816289fcfeb13cd1f871e72be2ea9b72f" gracePeriod=30
Sep 16 12:25:20.038703 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:20.038634    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:20.394394 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:20.394364    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:20.491393 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:20.491356    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:20.590056 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:20.590020    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:22.816409 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:22.816376    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:23.358353 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.358315    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:23.384863 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.384834    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"]
Sep 16 12:25:23.385099 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.385073    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p" podUID="d3145f96-225a-406c-b78a-411cec78fef8" containerName="payload-processing" containerID="cri-o://66be1904e6f831cd168148e3915ea4fa55c13a1233a930ef4d88fdab96f623c7" gracePeriod=30
Sep 16 12:25:23.392884 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.392854    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"]
Sep 16 12:25:23.393254 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.393230    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z" podUID="a71ce862-10f8-45f6-919d-17de2c394379" containerName="payload-pre-processing" containerID="cri-o://69b532d5b187eabf1f6b12efe26acdbc801ac81d892b97284eb02a8dc79dc315" gracePeriod=30
Sep 16 12:25:23.423514 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.423478    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:23.491772 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.491745    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:23.556339 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.556302    2558 generic.go:356] "Generic (PLEG): container finished" podID="a71ce862-10f8-45f6-919d-17de2c394379" containerID="69b532d5b187eabf1f6b12efe26acdbc801ac81d892b97284eb02a8dc79dc315" exitCode=0
Sep 16 12:25:23.556705 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.556475    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z" event={"ID":"a71ce862-10f8-45f6-919d-17de2c394379","Type":"ContainerDied","Data":"69b532d5b187eabf1f6b12efe26acdbc801ac81d892b97284eb02a8dc79dc315"}
Sep 16 12:25:23.559094 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.559068    2558 generic.go:356] "Generic (PLEG): container finished" podID="d3145f96-225a-406c-b78a-411cec78fef8" containerID="66be1904e6f831cd168148e3915ea4fa55c13a1233a930ef4d88fdab96f623c7" exitCode=0
Sep 16 12:25:23.559227 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.559151    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p" event={"ID":"d3145f96-225a-406c-b78a-411cec78fef8","Type":"ContainerDied","Data":"66be1904e6f831cd168148e3915ea4fa55c13a1233a930ef4d88fdab96f623c7"}
Sep 16 12:25:23.664521 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.664200    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"
Sep 16 12:25:23.699884 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.699852    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3145f96-225a-406c-b78a-411cec78fef8-plugins-config-volume\") pod \"d3145f96-225a-406c-b78a-411cec78fef8\" (UID: \"d3145f96-225a-406c-b78a-411cec78fef8\") "
Sep 16 12:25:23.700022 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.699929    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf55w\" (UniqueName: \"kubernetes.io/projected/d3145f96-225a-406c-b78a-411cec78fef8-kube-api-access-wf55w\") pod \"d3145f96-225a-406c-b78a-411cec78fef8\" (UID: \"d3145f96-225a-406c-b78a-411cec78fef8\") "
Sep 16 12:25:23.700286 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.700259    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3145f96-225a-406c-b78a-411cec78fef8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d3145f96-225a-406c-b78a-411cec78fef8" (UID: "d3145f96-225a-406c-b78a-411cec78fef8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:25:23.702280 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.702249    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3145f96-225a-406c-b78a-411cec78fef8-kube-api-access-wf55w" (OuterVolumeSpecName: "kube-api-access-wf55w") pod "d3145f96-225a-406c-b78a-411cec78fef8" (UID: "d3145f96-225a-406c-b78a-411cec78fef8"). InnerVolumeSpecName "kube-api-access-wf55w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:23.707775 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.707755    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"
Sep 16 12:25:23.800843 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.800811    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf4bz\" (UniqueName: \"kubernetes.io/projected/a71ce862-10f8-45f6-919d-17de2c394379-kube-api-access-cf4bz\") pod \"a71ce862-10f8-45f6-919d-17de2c394379\" (UID: \"a71ce862-10f8-45f6-919d-17de2c394379\") "
Sep 16 12:25:23.800994 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.800873    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71ce862-10f8-45f6-919d-17de2c394379-plugins-config-volume\") pod \"a71ce862-10f8-45f6-919d-17de2c394379\" (UID: \"a71ce862-10f8-45f6-919d-17de2c394379\") "
Sep 16 12:25:23.801116 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.801105    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3145f96-225a-406c-b78a-411cec78fef8-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:23.801152 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.801118    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf55w\" (UniqueName: \"kubernetes.io/projected/d3145f96-225a-406c-b78a-411cec78fef8-kube-api-access-wf55w\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:23.801269 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.801239    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71ce862-10f8-45f6-919d-17de2c394379-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a71ce862-10f8-45f6-919d-17de2c394379" (UID: "a71ce862-10f8-45f6-919d-17de2c394379"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:25:23.802882 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.802862    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71ce862-10f8-45f6-919d-17de2c394379-kube-api-access-cf4bz" (OuterVolumeSpecName: "kube-api-access-cf4bz") pod "a71ce862-10f8-45f6-919d-17de2c394379" (UID: "a71ce862-10f8-45f6-919d-17de2c394379"). InnerVolumeSpecName "kube-api-access-cf4bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:23.901694 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.901616    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71ce862-10f8-45f6-919d-17de2c394379-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:23.901694 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:23.901645    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf4bz\" (UniqueName: \"kubernetes.io/projected/a71ce862-10f8-45f6-919d-17de2c394379-kube-api-access-cf4bz\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:24.033186 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:24.033153    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:24.133097 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:24.133071    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:24.565533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:24.565505    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"
Sep 16 12:25:24.565735 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:24.565540    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p" event={"ID":"d3145f96-225a-406c-b78a-411cec78fef8","Type":"ContainerDied","Data":"5f67f8f28bc171c0987de39fe8389e5162db17421851334f7118fd15cc25380a"}
Sep 16 12:25:24.565735 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:24.565609    2558 scope.go:117] "RemoveContainer" containerID="66be1904e6f831cd168148e3915ea4fa55c13a1233a930ef4d88fdab96f623c7"
Sep 16 12:25:24.570253 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:24.569965    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z" event={"ID":"a71ce862-10f8-45f6-919d-17de2c394379","Type":"ContainerDied","Data":"9863b5ce581bf8c48a32c2daa8c0fdceae81cdb9cb86962e87a4e7ccd8d0aeee"}
Sep 16 12:25:24.570253 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:24.570059    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"
Sep 16 12:25:24.577339 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:24.577322    2558 scope.go:117] "RemoveContainer" containerID="69b532d5b187eabf1f6b12efe26acdbc801ac81d892b97284eb02a8dc79dc315"
Sep 16 12:25:24.596358 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:24.596336    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"]
Sep 16 12:25:24.615034 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:24.615013    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f1730c-64d6f885f7-4zm6p"]
Sep 16 12:25:24.647909 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:24.647884    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"]
Sep 16 12:25:24.662247 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:24.662190    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f1730c-849989c546-mdq9z"]
Sep 16 12:25:24.941826 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:24.941726    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:25.293451 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.293428    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:25:25.415966 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.415933    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-data\") pod \"65f71290-9ae1-4f1e-8b56-d13f69990f93\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") "
Sep 16 12:25:25.416160 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.415978    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-workload-socket\") pod \"65f71290-9ae1-4f1e-8b56-d13f69990f93\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") "
Sep 16 12:25:25.416160 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416002    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-workload-certs\") pod \"65f71290-9ae1-4f1e-8b56-d13f69990f93\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") "
Sep 16 12:25:25.416160 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416042    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f71290-9ae1-4f1e-8b56-d13f69990f93-istiod-ca-cert\") pod \"65f71290-9ae1-4f1e-8b56-d13f69990f93\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") "
Sep 16 12:25:25.416160 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416064    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-envoy\") pod \"65f71290-9ae1-4f1e-8b56-d13f69990f93\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") "
Sep 16 12:25:25.416160 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416111    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-credential-socket\") pod \"65f71290-9ae1-4f1e-8b56-d13f69990f93\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") "
Sep 16 12:25:25.416160 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416143    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpt2x\" (UniqueName: \"kubernetes.io/projected/65f71290-9ae1-4f1e-8b56-d13f69990f93-kube-api-access-hpt2x\") pod \"65f71290-9ae1-4f1e-8b56-d13f69990f93\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") "
Sep 16 12:25:25.416486 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416176    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-podinfo\") pod \"65f71290-9ae1-4f1e-8b56-d13f69990f93\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") "
Sep 16 12:25:25.416486 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416197    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-token\") pod \"65f71290-9ae1-4f1e-8b56-d13f69990f93\" (UID: \"65f71290-9ae1-4f1e-8b56-d13f69990f93\") "
Sep 16 12:25:25.416486 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416306    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65f71290-9ae1-4f1e-8b56-d13f69990f93" (UID: "65f71290-9ae1-4f1e-8b56-d13f69990f93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:25.416486 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416320    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-data" (OuterVolumeSpecName: "istio-data") pod "65f71290-9ae1-4f1e-8b56-d13f69990f93" (UID: "65f71290-9ae1-4f1e-8b56-d13f69990f93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:25.416486 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416415    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65f71290-9ae1-4f1e-8b56-d13f69990f93" (UID: "65f71290-9ae1-4f1e-8b56-d13f69990f93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:25.416795 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416616    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65f71290-9ae1-4f1e-8b56-d13f69990f93" (UID: "65f71290-9ae1-4f1e-8b56-d13f69990f93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:25.416795 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416627    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f71290-9ae1-4f1e-8b56-d13f69990f93-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65f71290-9ae1-4f1e-8b56-d13f69990f93" (UID: "65f71290-9ae1-4f1e-8b56-d13f69990f93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:25:25.416795 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416622    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-data\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:25.416795 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416669    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-workload-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:25.416795 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.416692    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-workload-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:25.418397 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.418374    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-token" (OuterVolumeSpecName: "istio-token") pod "65f71290-9ae1-4f1e-8b56-d13f69990f93" (UID: "65f71290-9ae1-4f1e-8b56-d13f69990f93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:25.418500 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.418434    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f71290-9ae1-4f1e-8b56-d13f69990f93-kube-api-access-hpt2x" (OuterVolumeSpecName: "kube-api-access-hpt2x") pod "65f71290-9ae1-4f1e-8b56-d13f69990f93" (UID: "65f71290-9ae1-4f1e-8b56-d13f69990f93"). InnerVolumeSpecName "kube-api-access-hpt2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:25.418783 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.418766    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65f71290-9ae1-4f1e-8b56-d13f69990f93" (UID: "65f71290-9ae1-4f1e-8b56-d13f69990f93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:25:25.418837 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.418784    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65f71290-9ae1-4f1e-8b56-d13f69990f93" (UID: "65f71290-9ae1-4f1e-8b56-d13f69990f93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:25.518376 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.518289    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-credential-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:25.518376 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.518330    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpt2x\" (UniqueName: \"kubernetes.io/projected/65f71290-9ae1-4f1e-8b56-d13f69990f93-kube-api-access-hpt2x\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:25.518376 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.518345    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-podinfo\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:25.518376 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.518358    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-token\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:25.518376 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.518369    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f71290-9ae1-4f1e-8b56-d13f69990f93-istiod-ca-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:25.518376 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.518381    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f71290-9ae1-4f1e-8b56-d13f69990f93-istio-envoy\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:25.578830 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.578798    2558 generic.go:356] "Generic (PLEG): container finished" podID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerID="0302e6568c954239e5a0501c7ccbb0f816289fcfeb13cd1f871e72be2ea9b72f" exitCode=0
Sep 16 12:25:25.579016 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.578856    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"
Sep 16 12:25:25.579016 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.578883    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm" event={"ID":"65f71290-9ae1-4f1e-8b56-d13f69990f93","Type":"ContainerDied","Data":"0302e6568c954239e5a0501c7ccbb0f816289fcfeb13cd1f871e72be2ea9b72f"}
Sep 16 12:25:25.579016 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.578939    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm" event={"ID":"65f71290-9ae1-4f1e-8b56-d13f69990f93","Type":"ContainerDied","Data":"5a717cf8f168433a392e87ae0fe88c1c55f36803bccf6e652aeea34a667f4be7"}
Sep 16 12:25:25.579016 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.578963    2558 scope.go:117] "RemoveContainer" containerID="0302e6568c954239e5a0501c7ccbb0f816289fcfeb13cd1f871e72be2ea9b72f"
Sep 16 12:25:25.588463 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.588448    2558 scope.go:117] "RemoveContainer" containerID="e3caf819a1cafdd6c558c1d71697d04da085cdeac59a1918cdfae1a4bd6deb1d"
Sep 16 12:25:25.595990 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.595970    2558 scope.go:117] "RemoveContainer" containerID="0302e6568c954239e5a0501c7ccbb0f816289fcfeb13cd1f871e72be2ea9b72f"
Sep 16 12:25:25.596239 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:25:25.596222    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0302e6568c954239e5a0501c7ccbb0f816289fcfeb13cd1f871e72be2ea9b72f\": container with ID starting with 0302e6568c954239e5a0501c7ccbb0f816289fcfeb13cd1f871e72be2ea9b72f not found: ID does not exist" containerID="0302e6568c954239e5a0501c7ccbb0f816289fcfeb13cd1f871e72be2ea9b72f"
Sep 16 12:25:25.596286 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.596247    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0302e6568c954239e5a0501c7ccbb0f816289fcfeb13cd1f871e72be2ea9b72f"} err="failed to get container status \"0302e6568c954239e5a0501c7ccbb0f816289fcfeb13cd1f871e72be2ea9b72f\": rpc error: code = NotFound desc = could not find container \"0302e6568c954239e5a0501c7ccbb0f816289fcfeb13cd1f871e72be2ea9b72f\": container with ID starting with 0302e6568c954239e5a0501c7ccbb0f816289fcfeb13cd1f871e72be2ea9b72f not found: ID does not exist"
Sep 16 12:25:25.596286 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.596264    2558 scope.go:117] "RemoveContainer" containerID="e3caf819a1cafdd6c558c1d71697d04da085cdeac59a1918cdfae1a4bd6deb1d"
Sep 16 12:25:25.596511 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:25:25.596489    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3caf819a1cafdd6c558c1d71697d04da085cdeac59a1918cdfae1a4bd6deb1d\": container with ID starting with e3caf819a1cafdd6c558c1d71697d04da085cdeac59a1918cdfae1a4bd6deb1d not found: ID does not exist" containerID="e3caf819a1cafdd6c558c1d71697d04da085cdeac59a1918cdfae1a4bd6deb1d"
Sep 16 12:25:25.596619 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.596515    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3caf819a1cafdd6c558c1d71697d04da085cdeac59a1918cdfae1a4bd6deb1d"} err="failed to get container status \"e3caf819a1cafdd6c558c1d71697d04da085cdeac59a1918cdfae1a4bd6deb1d\": rpc error: code = NotFound desc = could not find container \"e3caf819a1cafdd6c558c1d71697d04da085cdeac59a1918cdfae1a4bd6deb1d\": container with ID starting with e3caf819a1cafdd6c558c1d71697d04da085cdeac59a1918cdfae1a4bd6deb1d not found: ID does not exist"
Sep 16 12:25:25.603086 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.603067    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"]
Sep 16 12:25:25.608116 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.608094    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cadab1d8-openshift-default-6dbc8f7584-bx6vm"]
Sep 16 12:25:25.758880 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.758845    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f71290-9ae1-4f1e-8b56-d13f69990f93" path="/var/lib/kubelet/pods/65f71290-9ae1-4f1e-8b56-d13f69990f93/volumes"
Sep 16 12:25:25.759292 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.759277    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a71ce862-10f8-45f6-919d-17de2c394379" path="/var/lib/kubelet/pods/a71ce862-10f8-45f6-919d-17de2c394379/volumes"
Sep 16 12:25:25.759661 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:25.759646    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3145f96-225a-406c-b78a-411cec78fef8" path="/var/lib/kubelet/pods/d3145f96-225a-406c-b78a-411cec78fef8/volumes"
Sep 16 12:25:26.221743 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:26.221711    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:26.394594 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:26.394538    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:26.879973 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:26.879929    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:26.932454 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:26.932422    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:27.025353 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.025318    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:27.124707 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.124659    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:27.224791 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.224657    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:27.326044 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.326013    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:27.540656 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.540572    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs" podUID="b6614e64-c2c2-4985-a08a-52e37b88da92" containerName="main" containerID="cri-o://63fe38f5a48592767367c05b746186dce5cba71017412dc11c96a406475436b5" gracePeriod=45
Sep 16 12:25:27.785854 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.785829    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:27.845270 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.845247    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-dshm\") pod \"b6614e64-c2c2-4985-a08a-52e37b88da92\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") "
Sep 16 12:25:27.845419 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.845291    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6614e64-c2c2-4985-a08a-52e37b88da92-tls-certs\") pod \"b6614e64-c2c2-4985-a08a-52e37b88da92\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") "
Sep 16 12:25:27.845419 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.845323    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xptzd\" (UniqueName: \"kubernetes.io/projected/b6614e64-c2c2-4985-a08a-52e37b88da92-kube-api-access-xptzd\") pod \"b6614e64-c2c2-4985-a08a-52e37b88da92\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") "
Sep 16 12:25:27.845510 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.845425    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-home\") pod \"b6614e64-c2c2-4985-a08a-52e37b88da92\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") "
Sep 16 12:25:27.845510 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.845468    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-model-cache\") pod \"b6614e64-c2c2-4985-a08a-52e37b88da92\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") "
Sep 16 12:25:27.845864 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.845606    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-tmp-dir\") pod \"b6614e64-c2c2-4985-a08a-52e37b88da92\" (UID: \"b6614e64-c2c2-4985-a08a-52e37b88da92\") "
Sep 16 12:25:27.845864 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.845718    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6614e64-c2c2-4985-a08a-52e37b88da92" (UID: "b6614e64-c2c2-4985-a08a-52e37b88da92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:27.845864 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.845735    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-home" (OuterVolumeSpecName: "home") pod "b6614e64-c2c2-4985-a08a-52e37b88da92" (UID: "b6614e64-c2c2-4985-a08a-52e37b88da92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:27.845864 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.845849    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6614e64-c2c2-4985-a08a-52e37b88da92" (UID: "b6614e64-c2c2-4985-a08a-52e37b88da92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:27.846208 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.846186    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-home\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:27.846280 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.846213    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-model-cache\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:27.846280 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.846229    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-tmp-dir\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:27.847728 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.847702    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6614e64-c2c2-4985-a08a-52e37b88da92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6614e64-c2c2-4985-a08a-52e37b88da92" (UID: "b6614e64-c2c2-4985-a08a-52e37b88da92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:27.847728 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.847704    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6614e64-c2c2-4985-a08a-52e37b88da92-kube-api-access-xptzd" (OuterVolumeSpecName: "kube-api-access-xptzd") pod "b6614e64-c2c2-4985-a08a-52e37b88da92" (UID: "b6614e64-c2c2-4985-a08a-52e37b88da92"). InnerVolumeSpecName "kube-api-access-xptzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:27.847883 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.847792    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-dshm" (OuterVolumeSpecName: "dshm") pod "b6614e64-c2c2-4985-a08a-52e37b88da92" (UID: "b6614e64-c2c2-4985-a08a-52e37b88da92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:27.947629 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.947594    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6614e64-c2c2-4985-a08a-52e37b88da92-dshm\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:27.947629 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.947624    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6614e64-c2c2-4985-a08a-52e37b88da92-tls-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:27.947803 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:27.947639    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xptzd\" (UniqueName: \"kubernetes.io/projected/b6614e64-c2c2-4985-a08a-52e37b88da92-kube-api-access-xptzd\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:25:28.600072 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:28.600035    2558 generic.go:356] "Generic (PLEG): container finished" podID="b6614e64-c2c2-4985-a08a-52e37b88da92" containerID="63fe38f5a48592767367c05b746186dce5cba71017412dc11c96a406475436b5" exitCode=0
Sep 16 12:25:28.600533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:28.600081    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"
Sep 16 12:25:28.600533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:28.600118    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs" event={"ID":"b6614e64-c2c2-4985-a08a-52e37b88da92","Type":"ContainerDied","Data":"63fe38f5a48592767367c05b746186dce5cba71017412dc11c96a406475436b5"}
Sep 16 12:25:28.600533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:28.600164    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs" event={"ID":"b6614e64-c2c2-4985-a08a-52e37b88da92","Type":"ContainerDied","Data":"5bfa64c633f79342d86397acbc582eaa8b9ecfb3964cb20e378b528396a6ebc1"}
Sep 16 12:25:28.600533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:28.600181    2558 scope.go:117] "RemoveContainer" containerID="63fe38f5a48592767367c05b746186dce5cba71017412dc11c96a406475436b5"
Sep 16 12:25:28.609419 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:28.609402    2558 scope.go:117] "RemoveContainer" containerID="63fe38f5a48592767367c05b746186dce5cba71017412dc11c96a406475436b5"
Sep 16 12:25:28.609754 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:25:28.609711    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63fe38f5a48592767367c05b746186dce5cba71017412dc11c96a406475436b5\": container with ID starting with 63fe38f5a48592767367c05b746186dce5cba71017412dc11c96a406475436b5 not found: ID does not exist" containerID="63fe38f5a48592767367c05b746186dce5cba71017412dc11c96a406475436b5"
Sep 16 12:25:28.609806 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:28.609763    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63fe38f5a48592767367c05b746186dce5cba71017412dc11c96a406475436b5"} err="failed to get container status \"63fe38f5a48592767367c05b746186dce5cba71017412dc11c96a406475436b5\": rpc error: code = NotFound desc = could not find container \"63fe38f5a48592767367c05b746186dce5cba71017412dc11c96a406475436b5\": container with ID starting with 63fe38f5a48592767367c05b746186dce5cba71017412dc11c96a406475436b5 not found: ID does not exist"
Sep 16 12:25:28.621128 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:28.621090    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"]
Sep 16 12:25:28.630238 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:28.630214    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-f1730c/explicit-model-f1730c-kserve-7688dd85b9-zd9cs"]
Sep 16 12:25:29.749140 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:29.749100    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:29.758083 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:29.758057    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6614e64-c2c2-4985-a08a-52e37b88da92" path="/var/lib/kubelet/pods/b6614e64-c2c2-4985-a08a-52e37b88da92/volumes"
Sep 16 12:25:29.887017 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:29.886986    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:30.035701 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.035610    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:30.485782 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.485731    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:30.755231 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.755144    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"]
Sep 16 12:25:30.755815 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.755705    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerName="istio-proxy"
Sep 16 12:25:30.755900 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.755820    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerName="istio-proxy"
Sep 16 12:25:30.755900 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.755838    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6614e64-c2c2-4985-a08a-52e37b88da92" containerName="main"
Sep 16 12:25:30.755900 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.755847    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6614e64-c2c2-4985-a08a-52e37b88da92" containerName="main"
Sep 16 12:25:30.755900 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.755890    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71ce862-10f8-45f6-919d-17de2c394379" containerName="payload-pre-processing"
Sep 16 12:25:30.755900 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.755900    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71ce862-10f8-45f6-919d-17de2c394379" containerName="payload-pre-processing"
Sep 16 12:25:30.756169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.755909    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerName="istio-proxy"
Sep 16 12:25:30.756169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.755917    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerName="istio-proxy"
Sep 16 12:25:30.756169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.755951    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59216a8c-9e0b-4e43-bf95-c3f82e5610c6" containerName="payload-pre-processing"
Sep 16 12:25:30.756169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.755960    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="59216a8c-9e0b-4e43-bf95-c3f82e5610c6" containerName="payload-pre-processing"
Sep 16 12:25:30.756169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.755973    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3145f96-225a-406c-b78a-411cec78fef8" containerName="payload-processing"
Sep 16 12:25:30.756169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.755981    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3145f96-225a-406c-b78a-411cec78fef8" containerName="payload-processing"
Sep 16 12:25:30.756169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.756092    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="59216a8c-9e0b-4e43-bf95-c3f82e5610c6" containerName="payload-pre-processing"
Sep 16 12:25:30.756169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.756102    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6614e64-c2c2-4985-a08a-52e37b88da92" containerName="main"
Sep 16 12:25:30.756169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.756110    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerName="istio-proxy"
Sep 16 12:25:30.756169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.756117    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f71290-9ae1-4f1e-8b56-d13f69990f93" containerName="istio-proxy"
Sep 16 12:25:30.756169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.756124    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3145f96-225a-406c-b78a-411cec78fef8" containerName="payload-processing"
Sep 16 12:25:30.756169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.756132    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a71ce862-10f8-45f6-919d-17de2c394379" containerName="payload-pre-processing"
Sep 16 12:25:30.760541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.760521    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.763107 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.763088    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-52c0e5-openshift-default-dockercfg-c8gkk\""
Sep 16 12:25:30.773485 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.773463    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"]
Sep 16 12:25:30.783541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.783519    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:30.872751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.872716    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-data\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.872928 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.872806    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-credential-socket\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.872928 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.872854    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-workload-certs\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.872928 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.872908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-podinfo\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.873101 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.872963    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113bc108-afbd-4fab-ba5d-c84a754e87c5-istiod-ca-cert\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.873101 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.872992    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-workload-socket\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.873101 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.873009    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-envoy\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.873101 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.873074    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-token\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.873101 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.873099    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t4gw\" (UniqueName: \"kubernetes.io/projected/113bc108-afbd-4fab-ba5d-c84a754e87c5-kube-api-access-8t4gw\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.973558 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.973522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-workload-socket\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.973723 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.973581    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-envoy\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.973723 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.973612    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-token\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.973723 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.973639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t4gw\" (UniqueName: \"kubernetes.io/projected/113bc108-afbd-4fab-ba5d-c84a754e87c5-kube-api-access-8t4gw\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.973723 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.973668    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-data\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.973723 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.973697    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-credential-socket\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.974023 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.973734    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-workload-certs\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.974023 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.973882    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-podinfo\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.974023 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.973906    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-workload-socket\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.974023 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.974003    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113bc108-afbd-4fab-ba5d-c84a754e87c5-istiod-ca-cert\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.974224 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.974028    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-data\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.974224 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.974094    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-workload-certs\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.974224 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.974128    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-credential-socket\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.974627 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.974608    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113bc108-afbd-4fab-ba5d-c84a754e87c5-istiod-ca-cert\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.976020 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.975997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-envoy\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.976203 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.976186    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-podinfo\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.982085 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.982063    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-token\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:30.982174 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:30.982144    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t4gw\" (UniqueName: \"kubernetes.io/projected/113bc108-afbd-4fab-ba5d-c84a754e87c5-kube-api-access-8t4gw\") pod \"e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") " pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:31.071563 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:31.071470    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:31.195406 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:31.195362    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:31.366571 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:25:31.366512    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113bc108_afbd_4fab_ba5d_c84a754e87c5.slice/crio-cdeda568802851aaf7db549bb801aa9b518d2a00b46ff1421489eb59ce552d0a WatchSource:0}: Error finding container cdeda568802851aaf7db549bb801aa9b518d2a00b46ff1421489eb59ce552d0a: Status 404 returned error can't find the container with id cdeda568802851aaf7db549bb801aa9b518d2a00b46ff1421489eb59ce552d0a
Sep 16 12:25:31.368760 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:31.368726    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:25:31.368832 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:31.368794    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:25:31.368832 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:31.368822    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:25:31.370954 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:31.370929    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"]
Sep 16 12:25:31.619739 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:31.619526    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p" event={"ID":"113bc108-afbd-4fab-ba5d-c84a754e87c5","Type":"ContainerStarted","Data":"01b3f41eee7f944fe84edcaa8f2e0fc7bd2164c7d2baf10a14582f272dabea15"}
Sep 16 12:25:31.619739 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:31.619618    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p" event={"ID":"113bc108-afbd-4fab-ba5d-c84a754e87c5","Type":"ContainerStarted","Data":"cdeda568802851aaf7db549bb801aa9b518d2a00b46ff1421489eb59ce552d0a"}
Sep 16 12:25:31.643812 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:31.643757    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p" podStartSLOduration=1.643741076 podStartE2EDuration="1.643741076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:25:31.641257106 +0000 UTC m=+1328.395215836" watchObservedRunningTime="2026-09-16 12:25:31.643741076 +0000 UTC m=+1328.397699808"
Sep 16 12:25:31.937764 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:31.937685    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:32.038897 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:32.038865    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:32.072119 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:32.072084    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:32.077702 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:32.077677    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:32.142168 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:32.142135    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:32.235184 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:32.235104    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:32.626361 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:32.626329    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:32.627259 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:32.627242    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:25:33.672882 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:33.672854    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:33.774011 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:33.773970    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:34.606294 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:34.606260    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:34.873460 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:34.873365    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"]
Sep 16 12:25:34.878248 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:34.878218    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:34.880736 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:34.880710    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-cadd00-openshift-default-dockercfg-zxg94\""
Sep 16 12:25:34.915683 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:34.915650    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"]
Sep 16 12:25:34.917798 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:34.917767    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:35.016646 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.016609    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbrv9\" (UniqueName: \"kubernetes.io/projected/8df6ae1e-9b5c-482d-aa5e-b6d815938648-kube-api-access-gbrv9\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.016829 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.016712    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-podinfo\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.016829 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.016734    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-workload-socket\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.016829 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.016808    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-credential-socket\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.016966 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.016856    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-token\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.016966 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.016875    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istiod-ca-cert\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.016966 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.016898    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-workload-certs\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.016966 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.016924    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-envoy\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.017148 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.017029    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-data\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.117903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.117868    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbrv9\" (UniqueName: \"kubernetes.io/projected/8df6ae1e-9b5c-482d-aa5e-b6d815938648-kube-api-access-gbrv9\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.118088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.117934    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-podinfo\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.118088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.117954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-workload-socket\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.118088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.117983    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-credential-socket\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.118088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.118010    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-token\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.118088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.118032    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istiod-ca-cert\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.118088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.118057    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-workload-certs\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.118389 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.118091    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-envoy\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.118389 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.118132    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-data\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.118498 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.118467    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-credential-socket\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.118586 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.118517    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-workload-socket\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.118586 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.118532    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-data\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.118965 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.118930    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-workload-certs\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.118965 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.118981    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istiod-ca-cert\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.120371 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.120349    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-envoy\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.120472 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.120427    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-podinfo\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.125943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.125885    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbrv9\" (UniqueName: \"kubernetes.io/projected/8df6ae1e-9b5c-482d-aa5e-b6d815938648-kube-api-access-gbrv9\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.125943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.125930    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-token\") pod \"e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") " pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.137375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.137348    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:35.206496 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.206462    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:35.438815 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.438739    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:35.486595 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.486571    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"]
Sep 16 12:25:35.488534 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:25:35.488497    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df6ae1e_9b5c_482d_aa5e_b6d815938648.slice/crio-cf9055bffcb62ea05114fe70aa397220e93c7ea09f75f34afccf9d10d910cae9 WatchSource:0}: Error finding container cf9055bffcb62ea05114fe70aa397220e93c7ea09f75f34afccf9d10d910cae9: Status 404 returned error can't find the container with id cf9055bffcb62ea05114fe70aa397220e93c7ea09f75f34afccf9d10d910cae9
Sep 16 12:25:35.491078 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.491036    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:25:35.491187 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.491135    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:25:35.491187 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.491178    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:25:35.644705 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.644664    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" event={"ID":"8df6ae1e-9b5c-482d-aa5e-b6d815938648","Type":"ContainerStarted","Data":"841ba8b5c3fb1e37b553e947b08dc9b1346562098c46a0b3ced6f8350f1750b2"}
Sep 16 12:25:35.644869 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.644728    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" event={"ID":"8df6ae1e-9b5c-482d-aa5e-b6d815938648","Type":"ContainerStarted","Data":"cf9055bffcb62ea05114fe70aa397220e93c7ea09f75f34afccf9d10d910cae9"}
Sep 16 12:25:35.669863 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:35.669783    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" podStartSLOduration=1.669761472 podStartE2EDuration="1.669761472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:25:35.668342782 +0000 UTC m=+1332.422301513" watchObservedRunningTime="2026-09-16 12:25:35.669761472 +0000 UTC m=+1332.423720205"
Sep 16 12:25:36.207094 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:36.207056    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:36.212818 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:36.212788    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:36.649851 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:36.649815    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:36.650741 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:36.650719    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:25:36.944120 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:36.944044    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:37.545097 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:37.545064    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:37.647007 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:37.646972    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:37.744104 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:37.744070    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:38.551787 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:38.551755    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:39.471686 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:39.471649    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:40.141107 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:40.141070    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:41.133348 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.133317    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:41.189964 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.189935    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:41.269131 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.269102    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:41.567245 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.567165    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:41.697298 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.697267    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:41.791810 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.791778    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"]
Sep 16 12:25:41.794364 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.794343    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:41.795516 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.795492    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"
Sep 16 12:25:41.797572 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.797532    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-cadd00\""
Sep 16 12:25:41.806173 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.806152    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"]
Sep 16 12:25:41.838735 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.838659    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"]
Sep 16 12:25:41.842122 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.842107    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"
Sep 16 12:25:41.844191 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.844159    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-cadd00-dockercfg-78csh\""
Sep 16 12:25:41.850800 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.850773    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"]
Sep 16 12:25:41.988672 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.988634    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f99096b-29aa-4004-9879-3741bbe7797b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj\" (UID: \"5f99096b-29aa-4004-9879-3741bbe7797b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"
Sep 16 12:25:41.988840 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.988690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4q5\" (UniqueName: \"kubernetes.io/projected/560665e9-e61c-4b7a-86cd-54be23492507-kube-api-access-zt4q5\") pod \"payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2\" (UID: \"560665e9-e61c-4b7a-86cd-54be23492507\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"
Sep 16 12:25:41.988927 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.988900    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/560665e9-e61c-4b7a-86cd-54be23492507-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2\" (UID: \"560665e9-e61c-4b7a-86cd-54be23492507\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"
Sep 16 12:25:41.988979 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:41.988961    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxxn8\" (UniqueName: \"kubernetes.io/projected/5f99096b-29aa-4004-9879-3741bbe7797b-kube-api-access-gxxn8\") pod \"payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj\" (UID: \"5f99096b-29aa-4004-9879-3741bbe7797b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"
Sep 16 12:25:42.089743 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.089641    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/560665e9-e61c-4b7a-86cd-54be23492507-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2\" (UID: \"560665e9-e61c-4b7a-86cd-54be23492507\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"
Sep 16 12:25:42.089743 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.089687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxxn8\" (UniqueName: \"kubernetes.io/projected/5f99096b-29aa-4004-9879-3741bbe7797b-kube-api-access-gxxn8\") pod \"payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj\" (UID: \"5f99096b-29aa-4004-9879-3741bbe7797b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"
Sep 16 12:25:42.089743 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.089719    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f99096b-29aa-4004-9879-3741bbe7797b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj\" (UID: \"5f99096b-29aa-4004-9879-3741bbe7797b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"
Sep 16 12:25:42.089743 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.089740    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4q5\" (UniqueName: \"kubernetes.io/projected/560665e9-e61c-4b7a-86cd-54be23492507-kube-api-access-zt4q5\") pod \"payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2\" (UID: \"560665e9-e61c-4b7a-86cd-54be23492507\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"
Sep 16 12:25:42.090445 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.090421    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/560665e9-e61c-4b7a-86cd-54be23492507-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2\" (UID: \"560665e9-e61c-4b7a-86cd-54be23492507\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"
Sep 16 12:25:42.090580 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.090489    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f99096b-29aa-4004-9879-3741bbe7797b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj\" (UID: \"5f99096b-29aa-4004-9879-3741bbe7797b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"
Sep 16 12:25:42.097189 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.097160    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4q5\" (UniqueName: \"kubernetes.io/projected/560665e9-e61c-4b7a-86cd-54be23492507-kube-api-access-zt4q5\") pod \"payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2\" (UID: \"560665e9-e61c-4b7a-86cd-54be23492507\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"
Sep 16 12:25:42.097299 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.097280    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxxn8\" (UniqueName: \"kubernetes.io/projected/5f99096b-29aa-4004-9879-3741bbe7797b-kube-api-access-gxxn8\") pod \"payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj\" (UID: \"5f99096b-29aa-4004-9879-3741bbe7797b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"
Sep 16 12:25:42.108211 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.108192    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"
Sep 16 12:25:42.153738 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.153701    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"
Sep 16 12:25:42.380198 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.380172    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"]
Sep 16 12:25:42.381763 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:25:42.381735    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560665e9_e61c_4b7a_86cd_54be23492507.slice/crio-70d91bf6b1ce02511025bbb001ac6e6370806414839f8d9a9a74d9a556e58e15 WatchSource:0}: Error finding container 70d91bf6b1ce02511025bbb001ac6e6370806414839f8d9a9a74d9a556e58e15: Status 404 returned error can't find the container with id 70d91bf6b1ce02511025bbb001ac6e6370806414839f8d9a9a74d9a556e58e15
Sep 16 12:25:42.449089 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.449064    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"]
Sep 16 12:25:42.450376 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:25:42.450351    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f99096b_29aa_4004_9879_3741bbe7797b.slice/crio-5aa633b5cbf04fff7eb56d059e32f7bb909f22bede2663f7080451f497a4dffd WatchSource:0}: Error finding container 5aa633b5cbf04fff7eb56d059e32f7bb909f22bede2663f7080451f497a4dffd: Status 404 returned error can't find the container with id 5aa633b5cbf04fff7eb56d059e32f7bb909f22bede2663f7080451f497a4dffd
Sep 16 12:25:42.685201 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.685108    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj" event={"ID":"5f99096b-29aa-4004-9879-3741bbe7797b","Type":"ContainerStarted","Data":"644835d793fa9c5a2cedc1c3dcf8c3fd0f3f3b51bd6fd2c80f0990769986cae9"}
Sep 16 12:25:42.685201 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.685164    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj" event={"ID":"5f99096b-29aa-4004-9879-3741bbe7797b","Type":"ContainerStarted","Data":"5aa633b5cbf04fff7eb56d059e32f7bb909f22bede2663f7080451f497a4dffd"}
Sep 16 12:25:42.685424 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.685290    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"
Sep 16 12:25:42.686949 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.686919    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2" event={"ID":"560665e9-e61c-4b7a-86cd-54be23492507","Type":"ContainerStarted","Data":"d557f041253618f463fd146ce4aafba2a95771d5894d685f2244a534041959db"}
Sep 16 12:25:42.687082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.686969    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"
Sep 16 12:25:42.687082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.686986    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2" event={"ID":"560665e9-e61c-4b7a-86cd-54be23492507","Type":"ContainerStarted","Data":"70d91bf6b1ce02511025bbb001ac6e6370806414839f8d9a9a74d9a556e58e15"}
Sep 16 12:25:42.704009 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.703961    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj" podStartSLOduration=1.703942981 podStartE2EDuration="1.703942981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:25:42.701215968 +0000 UTC m=+1339.455174712" watchObservedRunningTime="2026-09-16 12:25:42.703942981 +0000 UTC m=+1339.457901713"
Sep 16 12:25:42.717667 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:42.717630    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2" podStartSLOduration=1.7176149120000002 podStartE2EDuration="1.717614912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:25:42.715153697 +0000 UTC m=+1339.469112430" watchObservedRunningTime="2026-09-16 12:25:42.717614912 +0000 UTC m=+1339.471573643"
Sep 16 12:25:51.811653 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:51.811561    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:53.695616 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:53.695580    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"
Sep 16 12:25:53.695999 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:53.695780    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"
Sep 16 12:25:54.164575 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:54.163864    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:57.646433 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:57.646399    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:57.738027 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:57.737991    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"]
Sep 16 12:25:57.738295 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:57.738267    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p" podUID="113bc108-afbd-4fab-ba5d-c84a754e87c5" containerName="istio-proxy" containerID="cri-o://01b3f41eee7f944fe84edcaa8f2e0fc7bd2164c7d2baf10a14582f272dabea15" gracePeriod=30
Sep 16 12:25:57.766270 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:57.766231    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:57.975514 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:57.975420    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:58.089998 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:58.089964    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:59.787360 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:59.787329    2558 generic.go:356] "Generic (PLEG): container finished" podID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerID="841ba8b5c3fb1e37b553e947b08dc9b1346562098c46a0b3ced6f8350f1750b2" exitCode=0
Sep 16 12:25:59.787805 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:59.787404    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" event={"ID":"8df6ae1e-9b5c-482d-aa5e-b6d815938648","Type":"ContainerDied","Data":"841ba8b5c3fb1e37b553e947b08dc9b1346562098c46a0b3ced6f8350f1750b2"}
Sep 16 12:25:59.787805 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:59.787792    2558 scope.go:117] "RemoveContainer" containerID="841ba8b5c3fb1e37b553e947b08dc9b1346562098c46a0b3ced6f8350f1750b2"
Sep 16 12:25:59.788954 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:59.788916    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:25:59.789046 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:59.789006    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:25:59.789087 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:25:59.789052    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:26:00.794891 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:00.794852    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" event={"ID":"8df6ae1e-9b5c-482d-aa5e-b6d815938648","Type":"ContainerStarted","Data":"8a25f3d723baaefb185da99ee0847784cc4aa577b1fbb92ba7a4ac8525c580f3"}
Sep 16 12:26:01.207419 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:01.207383    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:26:01.207688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:01.207666    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 12:26:01.207744 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:01.207713    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 12:26:02.206962 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:02.206927    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 12:26:02.207370 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:02.206982    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 12:26:02.241858 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:02.241820    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:02.628660 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:02.628615    2558 patch_prober.go:28] interesting pod/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 16 12:26:02.628863 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:02.628700    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p" podUID="113bc108-afbd-4fab-ba5d-c84a754e87c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:26:03.036023 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.035992    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:26:03.093915 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.093878    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-workload-socket\") pod \"113bc108-afbd-4fab-ba5d-c84a754e87c5\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") "
Sep 16 12:26:03.094122 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.093930    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t4gw\" (UniqueName: \"kubernetes.io/projected/113bc108-afbd-4fab-ba5d-c84a754e87c5-kube-api-access-8t4gw\") pod \"113bc108-afbd-4fab-ba5d-c84a754e87c5\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") "
Sep 16 12:26:03.094122 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.093970    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113bc108-afbd-4fab-ba5d-c84a754e87c5-istiod-ca-cert\") pod \"113bc108-afbd-4fab-ba5d-c84a754e87c5\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") "
Sep 16 12:26:03.094122 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.093997    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-credential-socket\") pod \"113bc108-afbd-4fab-ba5d-c84a754e87c5\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") "
Sep 16 12:26:03.094122 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.094085    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-data\") pod \"113bc108-afbd-4fab-ba5d-c84a754e87c5\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") "
Sep 16 12:26:03.094325 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.094127    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-workload-certs\") pod \"113bc108-afbd-4fab-ba5d-c84a754e87c5\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") "
Sep 16 12:26:03.094325 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.094156    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-envoy\") pod \"113bc108-afbd-4fab-ba5d-c84a754e87c5\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") "
Sep 16 12:26:03.094325 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.094153    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "113bc108-afbd-4fab-ba5d-c84a754e87c5" (UID: "113bc108-afbd-4fab-ba5d-c84a754e87c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:03.094325 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.094249    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-podinfo\") pod \"113bc108-afbd-4fab-ba5d-c84a754e87c5\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") "
Sep 16 12:26:03.094325 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.094290    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-token\") pod \"113bc108-afbd-4fab-ba5d-c84a754e87c5\" (UID: \"113bc108-afbd-4fab-ba5d-c84a754e87c5\") "
Sep 16 12:26:03.094325 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.094303    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "113bc108-afbd-4fab-ba5d-c84a754e87c5" (UID: "113bc108-afbd-4fab-ba5d-c84a754e87c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:03.094626 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.094529    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113bc108-afbd-4fab-ba5d-c84a754e87c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "113bc108-afbd-4fab-ba5d-c84a754e87c5" (UID: "113bc108-afbd-4fab-ba5d-c84a754e87c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:26:03.094626 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.094575    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "113bc108-afbd-4fab-ba5d-c84a754e87c5" (UID: "113bc108-afbd-4fab-ba5d-c84a754e87c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:03.094730 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.094639    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-data\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:03.094730 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.094659    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-workload-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:03.094730 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.094672    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113bc108-afbd-4fab-ba5d-c84a754e87c5-istiod-ca-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:03.094730 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.094686    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-credential-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:03.095224 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.095193    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "113bc108-afbd-4fab-ba5d-c84a754e87c5" (UID: "113bc108-afbd-4fab-ba5d-c84a754e87c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:03.096740 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.096711    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "113bc108-afbd-4fab-ba5d-c84a754e87c5" (UID: "113bc108-afbd-4fab-ba5d-c84a754e87c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:03.096841 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.096740    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113bc108-afbd-4fab-ba5d-c84a754e87c5-kube-api-access-8t4gw" (OuterVolumeSpecName: "kube-api-access-8t4gw") pod "113bc108-afbd-4fab-ba5d-c84a754e87c5" (UID: "113bc108-afbd-4fab-ba5d-c84a754e87c5"). InnerVolumeSpecName "kube-api-access-8t4gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:03.096916 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.096893    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "113bc108-afbd-4fab-ba5d-c84a754e87c5" (UID: "113bc108-afbd-4fab-ba5d-c84a754e87c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:03.099923 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.099895    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "113bc108-afbd-4fab-ba5d-c84a754e87c5" (UID: "113bc108-afbd-4fab-ba5d-c84a754e87c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:26:03.112390 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.112364    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:03.195413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.195318    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-workload-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:03.195413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.195349    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-envoy\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:03.195413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.195358    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-podinfo\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:03.195413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.195365    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113bc108-afbd-4fab-ba5d-c84a754e87c5-istio-token\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:03.195413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.195374    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t4gw\" (UniqueName: \"kubernetes.io/projected/113bc108-afbd-4fab-ba5d-c84a754e87c5-kube-api-access-8t4gw\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:03.207331 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.207300    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 12:26:03.207660 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.207358    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 12:26:03.573211 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.573114    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:03.812248 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.812221    2558 generic.go:356] "Generic (PLEG): container finished" podID="113bc108-afbd-4fab-ba5d-c84a754e87c5" containerID="01b3f41eee7f944fe84edcaa8f2e0fc7bd2164c7d2baf10a14582f272dabea15" exitCode=0
Sep 16 12:26:03.812400 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.812267    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"
Sep 16 12:26:03.812400 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.812314    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p" event={"ID":"113bc108-afbd-4fab-ba5d-c84a754e87c5","Type":"ContainerDied","Data":"01b3f41eee7f944fe84edcaa8f2e0fc7bd2164c7d2baf10a14582f272dabea15"}
Sep 16 12:26:03.812400 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.812363    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p" event={"ID":"113bc108-afbd-4fab-ba5d-c84a754e87c5","Type":"ContainerDied","Data":"cdeda568802851aaf7db549bb801aa9b518d2a00b46ff1421489eb59ce552d0a"}
Sep 16 12:26:03.812400 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.812385    2558 scope.go:117] "RemoveContainer" containerID="01b3f41eee7f944fe84edcaa8f2e0fc7bd2164c7d2baf10a14582f272dabea15"
Sep 16 12:26:03.821119 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.821099    2558 scope.go:117] "RemoveContainer" containerID="01b3f41eee7f944fe84edcaa8f2e0fc7bd2164c7d2baf10a14582f272dabea15"
Sep 16 12:26:03.821380 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:26:03.821361    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b3f41eee7f944fe84edcaa8f2e0fc7bd2164c7d2baf10a14582f272dabea15\": container with ID starting with 01b3f41eee7f944fe84edcaa8f2e0fc7bd2164c7d2baf10a14582f272dabea15 not found: ID does not exist" containerID="01b3f41eee7f944fe84edcaa8f2e0fc7bd2164c7d2baf10a14582f272dabea15"
Sep 16 12:26:03.821440 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.821387    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b3f41eee7f944fe84edcaa8f2e0fc7bd2164c7d2baf10a14582f272dabea15"} err="failed to get container status \"01b3f41eee7f944fe84edcaa8f2e0fc7bd2164c7d2baf10a14582f272dabea15\": rpc error: code = NotFound desc = could not find container \"01b3f41eee7f944fe84edcaa8f2e0fc7bd2164c7d2baf10a14582f272dabea15\": container with ID starting with 01b3f41eee7f944fe84edcaa8f2e0fc7bd2164c7d2baf10a14582f272dabea15 not found: ID does not exist"
Sep 16 12:26:03.828702 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.828646    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"]
Sep 16 12:26:03.832322 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:03.832303    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-52c0e5-openshift-default-7fcfc54f84-ml82p"]
Sep 16 12:26:04.207136 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:04.207091    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 12:26:04.207308 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:04.207160    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 12:26:04.435788 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:04.435740    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:05.207240 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:05.207202    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 12:26:05.207414 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:05.207262    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 12:26:05.246863 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:05.246833    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:05.763211 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:05.763174    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="113bc108-afbd-4fab-ba5d-c84a754e87c5" path="/var/lib/kubelet/pods/113bc108-afbd-4fab-ba5d-c84a754e87c5/volumes"
Sep 16 12:26:06.207346 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:06.207304    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 12:26:06.207531 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:06.207381    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 12:26:06.239399 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:06.239370    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:06.650301 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:06.650248    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:26:06.935492 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:06.935411    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:07.211434 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:07.211357    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:26:07.212264 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:07.212245    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:26:07.607037 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:07.607007    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:07.707185 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:07.707153    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:07.930369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:07.930288    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:08.239943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:08.239853    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:08.793369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:08.793316    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:10.182482 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:10.182449    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:10.910688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:10.910659    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:11.011201 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:11.011172    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:11.261066 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:11.260979    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:12.082207 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:12.082175    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:12.825154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:12.825119    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:13.431615 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.431584    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"]
Sep 16 12:26:13.432024 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.432009    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="113bc108-afbd-4fab-ba5d-c84a754e87c5" containerName="istio-proxy"
Sep 16 12:26:13.432092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.432026    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="113bc108-afbd-4fab-ba5d-c84a754e87c5" containerName="istio-proxy"
Sep 16 12:26:13.432131 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.432107    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="113bc108-afbd-4fab-ba5d-c84a754e87c5" containerName="istio-proxy"
Sep 16 12:26:13.435369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.435350    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.438322 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.438299    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"openshift-service-ca.crt\""
Sep 16 12:26:13.438435 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.438317    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"default-dockercfg-cwc4z\""
Sep 16 12:26:13.438435 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.438328    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"distinct-w0-739ee7-kserve-self-signed-certs\""
Sep 16 12:26:13.438435 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.438317    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"kube-root-ca.crt\""
Sep 16 12:26:13.443568 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.443522    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"]
Sep 16 12:26:13.503594 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.503559    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-dshm\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.503790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.503667    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-model-cache\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.503790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.503768    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-tmp-dir\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.503927 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.503872    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6x8c\" (UniqueName: \"kubernetes.io/projected/609f3195-96fa-45aa-842a-80222b4bf8ee-kube-api-access-r6x8c\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.503984 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.503910    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-home\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.504140 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.504122    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609f3195-96fa-45aa-842a-80222b4bf8ee-tls-certs\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.538247 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.538217    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:13.604932 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.604897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6x8c\" (UniqueName: \"kubernetes.io/projected/609f3195-96fa-45aa-842a-80222b4bf8ee-kube-api-access-r6x8c\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.605315 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.604941    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-home\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.605315 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.605040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609f3195-96fa-45aa-842a-80222b4bf8ee-tls-certs\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.605315 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.605077    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-dshm\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.605315 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.605106    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-model-cache\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.605315 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.605145    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-tmp-dir\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.605625 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.605425    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-home\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.605625 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.605473    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-model-cache\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.605625 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.605519    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-tmp-dir\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.607282 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.607260    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-dshm\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.607529 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.607513    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609f3195-96fa-45aa-842a-80222b4bf8ee-tls-certs\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.616463 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.616441    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6x8c\" (UniqueName: \"kubernetes.io/projected/609f3195-96fa-45aa-842a-80222b4bf8ee-kube-api-access-r6x8c\") pod \"distinct-w0-739ee7-kserve-767d687d55-nd957\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:13.638002 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.637974    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:13.745586 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:13.745484    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:14.029976 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:14.029948    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"]
Sep 16 12:26:14.031845 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:26:14.031820    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod609f3195_96fa_45aa_842a_80222b4bf8ee.slice/crio-766ebc018b04ca03a70ef1e1266eba5696436c1c528d0d1279b68ecbdd537902 WatchSource:0}: Error finding container 766ebc018b04ca03a70ef1e1266eba5696436c1c528d0d1279b68ecbdd537902: Status 404 returned error can't find the container with id 766ebc018b04ca03a70ef1e1266eba5696436c1c528d0d1279b68ecbdd537902
Sep 16 12:26:14.322951 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:14.322835    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:14.873597 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:14.873562    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957" event={"ID":"609f3195-96fa-45aa-842a-80222b4bf8ee","Type":"ContainerStarted","Data":"7255c788ace52c41427445001fb35a3ed1ba01a99601e71a5bcbb62d3356e507"}
Sep 16 12:26:14.873597 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:14.873602    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957" event={"ID":"609f3195-96fa-45aa-842a-80222b4bf8ee","Type":"ContainerStarted","Data":"766ebc018b04ca03a70ef1e1266eba5696436c1c528d0d1279b68ecbdd537902"}
Sep 16 12:26:14.898571 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:14.898499    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957" podStartSLOduration=1.89848393 podStartE2EDuration="1.89848393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:14.896300599 +0000 UTC m=+1371.650259332" watchObservedRunningTime="2026-09-16 12:26:14.89848393 +0000 UTC m=+1371.652442661"
Sep 16 12:26:15.142464 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.142388    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:15.207704 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.207666    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:15.328404 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.328372    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:15.603986 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.603959    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:15.701448 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.701401    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:15.734052 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.734023    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"]
Sep 16 12:26:15.738421 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.738399    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"
Sep 16 12:26:15.740501 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.740480    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-51ae50\""
Sep 16 12:26:15.740691 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.740661    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-51ae50-dockercfg-24r2w\""
Sep 16 12:26:15.746409 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.746382    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"]
Sep 16 12:26:15.829643 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.829598    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456f7c72-de2a-4092-bbc4-3db383337e2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr\" (UID: \"456f7c72-de2a-4092-bbc4-3db383337e2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"
Sep 16 12:26:15.829643 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.829641    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwtrk\" (UniqueName: \"kubernetes.io/projected/456f7c72-de2a-4092-bbc4-3db383337e2e-kube-api-access-vwtrk\") pod \"payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr\" (UID: \"456f7c72-de2a-4092-bbc4-3db383337e2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"
Sep 16 12:26:15.930584 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.930477    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456f7c72-de2a-4092-bbc4-3db383337e2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr\" (UID: \"456f7c72-de2a-4092-bbc4-3db383337e2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"
Sep 16 12:26:15.930584 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.930510    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwtrk\" (UniqueName: \"kubernetes.io/projected/456f7c72-de2a-4092-bbc4-3db383337e2e-kube-api-access-vwtrk\") pod \"payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr\" (UID: \"456f7c72-de2a-4092-bbc4-3db383337e2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"
Sep 16 12:26:15.931246 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.931221    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456f7c72-de2a-4092-bbc4-3db383337e2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr\" (UID: \"456f7c72-de2a-4092-bbc4-3db383337e2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"
Sep 16 12:26:15.939680 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:15.939655    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwtrk\" (UniqueName: \"kubernetes.io/projected/456f7c72-de2a-4092-bbc4-3db383337e2e-kube-api-access-vwtrk\") pod \"payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr\" (UID: \"456f7c72-de2a-4092-bbc4-3db383337e2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"
Sep 16 12:26:16.054412 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:16.054381    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"
Sep 16 12:26:16.327676 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:16.327645    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"]
Sep 16 12:26:16.329648 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:26:16.329621    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456f7c72_de2a_4092_bbc4_3db383337e2e.slice/crio-06d73d55bed80a98ad3f563e845f182d3497794beb05fcf4be9a08007d8037f9 WatchSource:0}: Error finding container 06d73d55bed80a98ad3f563e845f182d3497794beb05fcf4be9a08007d8037f9: Status 404 returned error can't find the container with id 06d73d55bed80a98ad3f563e845f182d3497794beb05fcf4be9a08007d8037f9
Sep 16 12:26:16.893376 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:16.893339    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr" event={"ID":"456f7c72-de2a-4092-bbc4-3db383337e2e","Type":"ContainerStarted","Data":"f65542eefbdf3e92b7e169ce6f9ce5952e0486e7c048ad174e2e1cdebd19f1f8"}
Sep 16 12:26:16.893603 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:16.893425    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr" event={"ID":"456f7c72-de2a-4092-bbc4-3db383337e2e","Type":"ContainerStarted","Data":"06d73d55bed80a98ad3f563e845f182d3497794beb05fcf4be9a08007d8037f9"}
Sep 16 12:26:16.893603 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:16.893572    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"
Sep 16 12:26:16.912491 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:16.912434    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr" podStartSLOduration=1.912415646 podStartE2EDuration="1.912415646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:16.908857953 +0000 UTC m=+1373.662816684" watchObservedRunningTime="2026-09-16 12:26:16.912415646 +0000 UTC m=+1373.666374378"
Sep 16 12:26:23.745853 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:23.745807    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:23.746330 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:23.745866    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:23.758735 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:23.758710    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:23.939403 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:23.939375    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:26:27.901072 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:27.901043    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"
Sep 16 12:26:37.073422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:37.073384    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:37.713294 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:37.713261    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:38.427120 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:38.427082    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:39.209408 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:39.209369    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:39.930983 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:39.930951    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:39.985620 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:39.985584    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"]
Sep 16 12:26:40.030912 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:40.030883    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:40.472401 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:40.472371    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:40.656140 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:40.656108    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:40.930534 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:40.930502    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:41.541327 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:41.541298    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:42.715375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:42.715348    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:42.819115 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:42.819081    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:50.382756 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:50.382719    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:50.652345 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:50.652265    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:51.119678 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:51.119631    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:53.523130 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:53.523085    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:54.164568 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:54.163923    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:54.268980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:54.268945    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:54.434369 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:54.434273    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:54.535133 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:54.535100    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:55.004560 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.004510    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm" podUID="172b8fc8-9fac-40c1-8900-543728756262" containerName="main" containerID="cri-o://2d58cb0e752bf0776c57da7c31f42dc231480fede430813880c8c6b6d2cbfb7a" gracePeriod=45
Sep 16 12:26:55.267963 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.267888    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:26:55.334791 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.334757    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:55.412107 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.412072    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-dshm\") pod \"172b8fc8-9fac-40c1-8900-543728756262\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") "
Sep 16 12:26:55.412107 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.412105    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-tmp-dir\") pod \"172b8fc8-9fac-40c1-8900-543728756262\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") "
Sep 16 12:26:55.412362 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.412142    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd4fx\" (UniqueName: \"kubernetes.io/projected/172b8fc8-9fac-40c1-8900-543728756262-kube-api-access-qd4fx\") pod \"172b8fc8-9fac-40c1-8900-543728756262\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") "
Sep 16 12:26:55.412362 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.412179    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-home\") pod \"172b8fc8-9fac-40c1-8900-543728756262\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") "
Sep 16 12:26:55.412362 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.412241    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-model-cache\") pod \"172b8fc8-9fac-40c1-8900-543728756262\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") "
Sep 16 12:26:55.412362 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.412301    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172b8fc8-9fac-40c1-8900-543728756262-tls-certs\") pod \"172b8fc8-9fac-40c1-8900-543728756262\" (UID: \"172b8fc8-9fac-40c1-8900-543728756262\") "
Sep 16 12:26:55.412580 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.412391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "172b8fc8-9fac-40c1-8900-543728756262" (UID: "172b8fc8-9fac-40c1-8900-543728756262"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:55.412580 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.412478    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-home" (OuterVolumeSpecName: "home") pod "172b8fc8-9fac-40c1-8900-543728756262" (UID: "172b8fc8-9fac-40c1-8900-543728756262"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:55.412580 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.412575    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-model-cache" (OuterVolumeSpecName: "model-cache") pod "172b8fc8-9fac-40c1-8900-543728756262" (UID: "172b8fc8-9fac-40c1-8900-543728756262"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:55.412794 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.412774    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-home\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:55.412794 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.412794    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-model-cache\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:55.412897 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.412804    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-tmp-dir\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:55.414367 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.414341    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-dshm" (OuterVolumeSpecName: "dshm") pod "172b8fc8-9fac-40c1-8900-543728756262" (UID: "172b8fc8-9fac-40c1-8900-543728756262"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:55.414466 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.414423    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/172b8fc8-9fac-40c1-8900-543728756262-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "172b8fc8-9fac-40c1-8900-543728756262" (UID: "172b8fc8-9fac-40c1-8900-543728756262"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:26:55.414573 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.414531    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172b8fc8-9fac-40c1-8900-543728756262-kube-api-access-qd4fx" (OuterVolumeSpecName: "kube-api-access-qd4fx") pod "172b8fc8-9fac-40c1-8900-543728756262" (UID: "172b8fc8-9fac-40c1-8900-543728756262"). InnerVolumeSpecName "kube-api-access-qd4fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:55.514011 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.513977    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172b8fc8-9fac-40c1-8900-543728756262-tls-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:55.514011 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.514004    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172b8fc8-9fac-40c1-8900-543728756262-dshm\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:55.514011 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:55.514013    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd4fx\" (UniqueName: \"kubernetes.io/projected/172b8fc8-9fac-40c1-8900-543728756262-kube-api-access-qd4fx\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:26:56.099614 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:56.099570    2558 generic.go:356] "Generic (PLEG): container finished" podID="172b8fc8-9fac-40c1-8900-543728756262" containerID="2d58cb0e752bf0776c57da7c31f42dc231480fede430813880c8c6b6d2cbfb7a" exitCode=0
Sep 16 12:26:56.099614 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:56.099611    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"
Sep 16 12:26:56.100104 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:56.099606    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm" event={"ID":"172b8fc8-9fac-40c1-8900-543728756262","Type":"ContainerDied","Data":"2d58cb0e752bf0776c57da7c31f42dc231480fede430813880c8c6b6d2cbfb7a"}
Sep 16 12:26:56.100104 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:56.099727    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm" event={"ID":"172b8fc8-9fac-40c1-8900-543728756262","Type":"ContainerDied","Data":"5b4495f7c8700dc76fe22016fc808771ce8a534dd01d281784d3531efe4f2b08"}
Sep 16 12:26:56.100104 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:56.099750    2558 scope.go:117] "RemoveContainer" containerID="2d58cb0e752bf0776c57da7c31f42dc231480fede430813880c8c6b6d2cbfb7a"
Sep 16 12:26:56.108705 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:56.108686    2558 scope.go:117] "RemoveContainer" containerID="2d58cb0e752bf0776c57da7c31f42dc231480fede430813880c8c6b6d2cbfb7a"
Sep 16 12:26:56.108993 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:26:56.108968    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d58cb0e752bf0776c57da7c31f42dc231480fede430813880c8c6b6d2cbfb7a\": container with ID starting with 2d58cb0e752bf0776c57da7c31f42dc231480fede430813880c8c6b6d2cbfb7a not found: ID does not exist" containerID="2d58cb0e752bf0776c57da7c31f42dc231480fede430813880c8c6b6d2cbfb7a"
Sep 16 12:26:56.109114 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:56.109004    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d58cb0e752bf0776c57da7c31f42dc231480fede430813880c8c6b6d2cbfb7a"} err="failed to get container status \"2d58cb0e752bf0776c57da7c31f42dc231480fede430813880c8c6b6d2cbfb7a\": rpc error: code = NotFound desc = could not find container \"2d58cb0e752bf0776c57da7c31f42dc231480fede430813880c8c6b6d2cbfb7a\": container with ID starting with 2d58cb0e752bf0776c57da7c31f42dc231480fede430813880c8c6b6d2cbfb7a not found: ID does not exist"
Sep 16 12:26:56.116502 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:56.116476    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"]
Sep 16 12:26:56.120278 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:56.120253    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f79e22/test-model-f79e22-kserve-56555cd7ff-hw9qm"]
Sep 16 12:26:56.229599 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:56.229568    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:56.880086 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:56.880054    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:57.643448 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:57.643419    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:57.758212 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:57.758180    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="172b8fc8-9fac-40c1-8900-543728756262" path="/var/lib/kubelet/pods/172b8fc8-9fac-40c1-8900-543728756262/volumes"
Sep 16 12:26:58.336423 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:58.336393    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:59.014795 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.014764    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:59.120096 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.120057    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:59.477031 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.477003    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:59.583706 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.583672    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:59.611391 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.611351    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"]
Sep 16 12:26:59.611877 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.611853    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="172b8fc8-9fac-40c1-8900-543728756262" containerName="main"
Sep 16 12:26:59.611877 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.611873    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="172b8fc8-9fac-40c1-8900-543728756262" containerName="main"
Sep 16 12:26:59.612066 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.611951    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="172b8fc8-9fac-40c1-8900-543728756262" containerName="main"
Sep 16 12:26:59.616446 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.616423    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"
Sep 16 12:26:59.618820 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.618802    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d5891124\""
Sep 16 12:26:59.618925 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.618806    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d5891124-dockercfg-7h7zx\""
Sep 16 12:26:59.624965 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.624943    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"]
Sep 16 12:26:59.753586 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.753464    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kpb8\" (UniqueName: \"kubernetes.io/projected/acd69949-347b-40b7-819f-7a99e7042700-kube-api-access-9kpb8\") pod \"payload-processing-e2e-mt-d5891124-6bbf985965-z2lht\" (UID: \"acd69949-347b-40b7-819f-7a99e7042700\") " pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"
Sep 16 12:26:59.753586 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.753523    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acd69949-347b-40b7-819f-7a99e7042700-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5891124-6bbf985965-z2lht\" (UID: \"acd69949-347b-40b7-819f-7a99e7042700\") " pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"
Sep 16 12:26:59.830601 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.830513    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:59.854700 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.854664    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kpb8\" (UniqueName: \"kubernetes.io/projected/acd69949-347b-40b7-819f-7a99e7042700-kube-api-access-9kpb8\") pod \"payload-processing-e2e-mt-d5891124-6bbf985965-z2lht\" (UID: \"acd69949-347b-40b7-819f-7a99e7042700\") " pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"
Sep 16 12:26:59.854888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.854711    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acd69949-347b-40b7-819f-7a99e7042700-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5891124-6bbf985965-z2lht\" (UID: \"acd69949-347b-40b7-819f-7a99e7042700\") " pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"
Sep 16 12:26:59.855506 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.855478    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acd69949-347b-40b7-819f-7a99e7042700-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5891124-6bbf985965-z2lht\" (UID: \"acd69949-347b-40b7-819f-7a99e7042700\") " pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"
Sep 16 12:26:59.863044 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.863016    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kpb8\" (UniqueName: \"kubernetes.io/projected/acd69949-347b-40b7-819f-7a99e7042700-kube-api-access-9kpb8\") pod \"payload-processing-e2e-mt-d5891124-6bbf985965-z2lht\" (UID: \"acd69949-347b-40b7-819f-7a99e7042700\") " pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"
Sep 16 12:26:59.928514 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:26:59.928480    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"
Sep 16 12:27:00.202164 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:00.202140    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"]
Sep 16 12:27:00.204452 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:27:00.204421    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd69949_347b_40b7_819f_7a99e7042700.slice/crio-3b1c87037ab2c21d4997df321faf72b8c64eaa811f5816bb145e1ad0a7530815 WatchSource:0}: Error finding container 3b1c87037ab2c21d4997df321faf72b8c64eaa811f5816bb145e1ad0a7530815: Status 404 returned error can't find the container with id 3b1c87037ab2c21d4997df321faf72b8c64eaa811f5816bb145e1ad0a7530815
Sep 16 12:27:01.136039 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:01.135995    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht" event={"ID":"acd69949-347b-40b7-819f-7a99e7042700","Type":"ContainerStarted","Data":"3227fc57e9b5d1439a62cc834f03e6b3c0ab24ac718b8b6dad339f2f00abe8b2"}
Sep 16 12:27:01.136232 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:01.136067    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht" event={"ID":"acd69949-347b-40b7-819f-7a99e7042700","Type":"ContainerStarted","Data":"3b1c87037ab2c21d4997df321faf72b8c64eaa811f5816bb145e1ad0a7530815"}
Sep 16 12:27:01.136232 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:01.136180    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"
Sep 16 12:27:01.152684 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:01.152621    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht" podStartSLOduration=2.152602749 podStartE2EDuration="2.152602749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:27:01.152066596 +0000 UTC m=+1417.906025340" watchObservedRunningTime="2026-09-16 12:27:01.152602749 +0000 UTC m=+1417.906561482"
Sep 16 12:27:06.404390 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:06.404347    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:07.201375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:07.201341    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:07.902283 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:07.902251    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:08.259613 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:08.259493    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:08.572097 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:08.572018    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:09.336402 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:09.336365    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:09.967189 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:09.967140    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:10.846778 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:10.846747    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:11.801766 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:11.801733    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:12.144419 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:12.144388    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"
Sep 16 12:27:12.343410 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:12.343371    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:13.115893 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:13.115861    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:13.985354 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:13.985322    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:14.088909 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:14.088867    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:14.822981 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:14.822946    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:15.147842 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:15.147799    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:15.703007 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:15.702974    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:16.916441 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:16.916412    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:17.555273 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:17.555239    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:18.233025 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:18.232995    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:18.788939 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:18.788907    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"]
Sep 16 12:27:18.789167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:18.789144    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq" podUID="b4b1f827-a3be-4282-b03a-2132d3b62edd" containerName="maas-api" containerID="cri-o://12d1151f3663e1822834efd4aa60c7de071fd5cd5981a18b026533f82315a8de" gracePeriod=30
Sep 16 12:27:18.812978 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:18.812952    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:18.843120 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:18.843092    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"]
Sep 16 12:27:18.843331 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:18.843307    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk" podUID="fb135408-2e81-4cca-9522-92d16cddb49d" containerName="payload-processing" containerID="cri-o://892aaad2350ca858d9d8edc4f93586d522d52a10afad729becf57e565e00bcce" gracePeriod=30
Sep 16 12:27:18.912812 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:18.912760    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:19.017162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.017111    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:19.083939 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.083907    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:19.092602 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.092577    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:27:19.135541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.135389    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"
Sep 16 12:27:19.145010 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.144725    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b1f827-a3be-4282-b03a-2132d3b62edd-openshift-service-ca\") pod \"b4b1f827-a3be-4282-b03a-2132d3b62edd\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") "
Sep 16 12:27:19.145010 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.144834    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4b1f827-a3be-4282-b03a-2132d3b62edd-metrics-certs\") pod \"b4b1f827-a3be-4282-b03a-2132d3b62edd\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") "
Sep 16 12:27:19.145010 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.144890    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4b1f827-a3be-4282-b03a-2132d3b62edd-maas-api-tls\") pod \"b4b1f827-a3be-4282-b03a-2132d3b62edd\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") "
Sep 16 12:27:19.145010 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.144925    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6rd4\" (UniqueName: \"kubernetes.io/projected/b4b1f827-a3be-4282-b03a-2132d3b62edd-kube-api-access-g6rd4\") pod \"b4b1f827-a3be-4282-b03a-2132d3b62edd\" (UID: \"b4b1f827-a3be-4282-b03a-2132d3b62edd\") "
Sep 16 12:27:19.147540 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.146313    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b1f827-a3be-4282-b03a-2132d3b62edd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b4b1f827-a3be-4282-b03a-2132d3b62edd" (UID: "b4b1f827-a3be-4282-b03a-2132d3b62edd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:19.148275 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.148236    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b1f827-a3be-4282-b03a-2132d3b62edd-kube-api-access-g6rd4" (OuterVolumeSpecName: "kube-api-access-g6rd4") pod "b4b1f827-a3be-4282-b03a-2132d3b62edd" (UID: "b4b1f827-a3be-4282-b03a-2132d3b62edd"). InnerVolumeSpecName "kube-api-access-g6rd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:19.149188 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.149162    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b1f827-a3be-4282-b03a-2132d3b62edd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b4b1f827-a3be-4282-b03a-2132d3b62edd" (UID: "b4b1f827-a3be-4282-b03a-2132d3b62edd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:19.149717 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.149683    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b1f827-a3be-4282-b03a-2132d3b62edd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b4b1f827-a3be-4282-b03a-2132d3b62edd" (UID: "b4b1f827-a3be-4282-b03a-2132d3b62edd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:19.239828 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.239791    2558 generic.go:356] "Generic (PLEG): container finished" podID="fb135408-2e81-4cca-9522-92d16cddb49d" containerID="892aaad2350ca858d9d8edc4f93586d522d52a10afad729becf57e565e00bcce" exitCode=0
Sep 16 12:27:19.240289 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.239853    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"
Sep 16 12:27:19.240289 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.239875    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk" event={"ID":"fb135408-2e81-4cca-9522-92d16cddb49d","Type":"ContainerDied","Data":"892aaad2350ca858d9d8edc4f93586d522d52a10afad729becf57e565e00bcce"}
Sep 16 12:27:19.240289 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.239948    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk" event={"ID":"fb135408-2e81-4cca-9522-92d16cddb49d","Type":"ContainerDied","Data":"3fafdebdf464b5d37aa40fecfaf0af710e3b21bd20a64b6c02ddd89bfcc425ce"}
Sep 16 12:27:19.240289 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.239978    2558 scope.go:117] "RemoveContainer" containerID="892aaad2350ca858d9d8edc4f93586d522d52a10afad729becf57e565e00bcce"
Sep 16 12:27:19.241425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.241404    2558 generic.go:356] "Generic (PLEG): container finished" podID="b4b1f827-a3be-4282-b03a-2132d3b62edd" containerID="12d1151f3663e1822834efd4aa60c7de071fd5cd5981a18b026533f82315a8de" exitCode=0
Sep 16 12:27:19.241541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.241452    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"
Sep 16 12:27:19.241541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.241473    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq" event={"ID":"b4b1f827-a3be-4282-b03a-2132d3b62edd","Type":"ContainerDied","Data":"12d1151f3663e1822834efd4aa60c7de071fd5cd5981a18b026533f82315a8de"}
Sep 16 12:27:19.241541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.241508    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq" event={"ID":"b4b1f827-a3be-4282-b03a-2132d3b62edd","Type":"ContainerDied","Data":"b03a1680438306c3f72a76100cc5dfc47d81e28cb00687a853b30c2f3cdd6f97"}
Sep 16 12:27:19.245534 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.245514    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tntkq\" (UniqueName: \"kubernetes.io/projected/fb135408-2e81-4cca-9522-92d16cddb49d-kube-api-access-tntkq\") pod \"fb135408-2e81-4cca-9522-92d16cddb49d\" (UID: \"fb135408-2e81-4cca-9522-92d16cddb49d\") "
Sep 16 12:27:19.245689 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.245671    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb135408-2e81-4cca-9522-92d16cddb49d-plugins-config-volume\") pod \"fb135408-2e81-4cca-9522-92d16cddb49d\" (UID: \"fb135408-2e81-4cca-9522-92d16cddb49d\") "
Sep 16 12:27:19.246081 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.246023    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb135408-2e81-4cca-9522-92d16cddb49d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb135408-2e81-4cca-9522-92d16cddb49d" (UID: "fb135408-2e81-4cca-9522-92d16cddb49d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:19.246361 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.246339    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4b1f827-a3be-4282-b03a-2132d3b62edd-metrics-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:27:19.246445 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.246366    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4b1f827-a3be-4282-b03a-2132d3b62edd-maas-api-tls\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:27:19.246445 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.246382    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6rd4\" (UniqueName: \"kubernetes.io/projected/b4b1f827-a3be-4282-b03a-2132d3b62edd-kube-api-access-g6rd4\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:27:19.246445 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.246435    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb135408-2e81-4cca-9522-92d16cddb49d-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:27:19.246623 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.246452    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b1f827-a3be-4282-b03a-2132d3b62edd-openshift-service-ca\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:27:19.247794 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.247772    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb135408-2e81-4cca-9522-92d16cddb49d-kube-api-access-tntkq" (OuterVolumeSpecName: "kube-api-access-tntkq") pod "fb135408-2e81-4cca-9522-92d16cddb49d" (UID: "fb135408-2e81-4cca-9522-92d16cddb49d"). InnerVolumeSpecName "kube-api-access-tntkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:19.249303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.249288    2558 scope.go:117] "RemoveContainer" containerID="892aaad2350ca858d9d8edc4f93586d522d52a10afad729becf57e565e00bcce"
Sep 16 12:27:19.249613 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:27:19.249589    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"892aaad2350ca858d9d8edc4f93586d522d52a10afad729becf57e565e00bcce\": container with ID starting with 892aaad2350ca858d9d8edc4f93586d522d52a10afad729becf57e565e00bcce not found: ID does not exist" containerID="892aaad2350ca858d9d8edc4f93586d522d52a10afad729becf57e565e00bcce"
Sep 16 12:27:19.249700 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.249617    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"892aaad2350ca858d9d8edc4f93586d522d52a10afad729becf57e565e00bcce"} err="failed to get container status \"892aaad2350ca858d9d8edc4f93586d522d52a10afad729becf57e565e00bcce\": rpc error: code = NotFound desc = could not find container \"892aaad2350ca858d9d8edc4f93586d522d52a10afad729becf57e565e00bcce\": container with ID starting with 892aaad2350ca858d9d8edc4f93586d522d52a10afad729becf57e565e00bcce not found: ID does not exist"
Sep 16 12:27:19.249700 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.249648    2558 scope.go:117] "RemoveContainer" containerID="12d1151f3663e1822834efd4aa60c7de071fd5cd5981a18b026533f82315a8de"
Sep 16 12:27:19.259754 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.259731    2558 scope.go:117] "RemoveContainer" containerID="12d1151f3663e1822834efd4aa60c7de071fd5cd5981a18b026533f82315a8de"
Sep 16 12:27:19.260055 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:27:19.260033    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d1151f3663e1822834efd4aa60c7de071fd5cd5981a18b026533f82315a8de\": container with ID starting with 12d1151f3663e1822834efd4aa60c7de071fd5cd5981a18b026533f82315a8de not found: ID does not exist" containerID="12d1151f3663e1822834efd4aa60c7de071fd5cd5981a18b026533f82315a8de"
Sep 16 12:27:19.260165 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.260061    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d1151f3663e1822834efd4aa60c7de071fd5cd5981a18b026533f82315a8de"} err="failed to get container status \"12d1151f3663e1822834efd4aa60c7de071fd5cd5981a18b026533f82315a8de\": rpc error: code = NotFound desc = could not find container \"12d1151f3663e1822834efd4aa60c7de071fd5cd5981a18b026533f82315a8de\": container with ID starting with 12d1151f3663e1822834efd4aa60c7de071fd5cd5981a18b026533f82315a8de not found: ID does not exist"
Sep 16 12:27:19.264672 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.264640    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"]
Sep 16 12:27:19.266808 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.266787    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f79e22-68b898898c-6kjpq"]
Sep 16 12:27:19.347928 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.347901    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tntkq\" (UniqueName: \"kubernetes.io/projected/fb135408-2e81-4cca-9522-92d16cddb49d-kube-api-access-tntkq\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:27:19.383541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.383513    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:19.489857 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.489761    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:19.561370 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.561344    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"]
Sep 16 12:27:19.563179 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.563159    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f79e22-d7bb99c67-jdhwk"]
Sep 16 12:27:19.758542 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.758458    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4b1f827-a3be-4282-b03a-2132d3b62edd" path="/var/lib/kubelet/pods/b4b1f827-a3be-4282-b03a-2132d3b62edd/volumes"
Sep 16 12:27:19.758879 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:19.758863    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb135408-2e81-4cca-9522-92d16cddb49d" path="/var/lib/kubelet/pods/fb135408-2e81-4cca-9522-92d16cddb49d/volumes"
Sep 16 12:27:20.069027 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:20.068947    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:20.744233 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:20.744201    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:21.602214 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:21.602164    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:22.256057 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:22.256026    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:22.888806 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:22.888777    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:23.583263 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:23.583230    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:23.808320 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:23.808289    2558 scope.go:117] "RemoveContainer" containerID="5dc7b5d3f2c7908f3b51f953482f3844195e1f9fdd3239a07b1528eb8d39236a"
Sep 16 12:27:25.489051 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:25.489017    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:25.624243 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:25.624210    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:25.781094 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:25.781009    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:26.401796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:26.401759    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:27.141649 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:27.141614    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:27.792604 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:27.792574    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:28.436566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:28.436512    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:29.179026 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:29.178990    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:29.830912 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:29.830876    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:30.492786 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:30.492756    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:31.165688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:31.165655    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:36.635801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:36.635759    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"]
Sep 16 12:27:36.636272 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:36.636064    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c" podUID="453f7dfa-4456-4a99-b2e3-6d4852d9d3ba" containerName="maas-api" containerID="cri-o://fd928d778461259ddc6b2ad5db79730cf4cac7d6f95c93d286982b58ef1acd7a" gracePeriod=30
Sep 16 12:27:36.880208 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:36.880181    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:27:37.016338 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.016241    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-maas-api-tls\") pod \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") "
Sep 16 12:27:37.016504 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.016355    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrw7c\" (UniqueName: \"kubernetes.io/projected/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-kube-api-access-qrw7c\") pod \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") "
Sep 16 12:27:37.016504 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.016451    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-metrics-certs\") pod \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") "
Sep 16 12:27:37.016669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.016522    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-openshift-service-ca\") pod \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\" (UID: \"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba\") "
Sep 16 12:27:37.017240 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.017208    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "453f7dfa-4456-4a99-b2e3-6d4852d9d3ba" (UID: "453f7dfa-4456-4a99-b2e3-6d4852d9d3ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:37.019141 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.019115    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-kube-api-access-qrw7c" (OuterVolumeSpecName: "kube-api-access-qrw7c") pod "453f7dfa-4456-4a99-b2e3-6d4852d9d3ba" (UID: "453f7dfa-4456-4a99-b2e3-6d4852d9d3ba"). InnerVolumeSpecName "kube-api-access-qrw7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:37.019423 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.019405    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "453f7dfa-4456-4a99-b2e3-6d4852d9d3ba" (UID: "453f7dfa-4456-4a99-b2e3-6d4852d9d3ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:37.019583 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.019537    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "453f7dfa-4456-4a99-b2e3-6d4852d9d3ba" (UID: "453f7dfa-4456-4a99-b2e3-6d4852d9d3ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:37.118213 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.118177    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrw7c\" (UniqueName: \"kubernetes.io/projected/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-kube-api-access-qrw7c\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:27:37.118213 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.118208    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-metrics-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:27:37.118213 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.118218    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-openshift-service-ca\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:27:37.118449 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.118228    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba-maas-api-tls\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:27:37.338267 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.338167    2558 generic.go:356] "Generic (PLEG): container finished" podID="453f7dfa-4456-4a99-b2e3-6d4852d9d3ba" containerID="fd928d778461259ddc6b2ad5db79730cf4cac7d6f95c93d286982b58ef1acd7a" exitCode=0
Sep 16 12:27:37.338267 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.338206    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c" event={"ID":"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba","Type":"ContainerDied","Data":"fd928d778461259ddc6b2ad5db79730cf4cac7d6f95c93d286982b58ef1acd7a"}
Sep 16 12:27:37.338267 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.338233    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"
Sep 16 12:27:37.338267 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.338244    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c" event={"ID":"453f7dfa-4456-4a99-b2e3-6d4852d9d3ba","Type":"ContainerDied","Data":"41be26cc7b499da36e1f7c3859d1b4c263086596166dfd7f593e978716761bb2"}
Sep 16 12:27:37.338267 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.338268    2558 scope.go:117] "RemoveContainer" containerID="fd928d778461259ddc6b2ad5db79730cf4cac7d6f95c93d286982b58ef1acd7a"
Sep 16 12:27:37.347778 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.347760    2558 scope.go:117] "RemoveContainer" containerID="fd928d778461259ddc6b2ad5db79730cf4cac7d6f95c93d286982b58ef1acd7a"
Sep 16 12:27:37.348034 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:27:37.348017    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd928d778461259ddc6b2ad5db79730cf4cac7d6f95c93d286982b58ef1acd7a\": container with ID starting with fd928d778461259ddc6b2ad5db79730cf4cac7d6f95c93d286982b58ef1acd7a not found: ID does not exist" containerID="fd928d778461259ddc6b2ad5db79730cf4cac7d6f95c93d286982b58ef1acd7a"
Sep 16 12:27:37.348081 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.348039    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd928d778461259ddc6b2ad5db79730cf4cac7d6f95c93d286982b58ef1acd7a"} err="failed to get container status \"fd928d778461259ddc6b2ad5db79730cf4cac7d6f95c93d286982b58ef1acd7a\": rpc error: code = NotFound desc = could not find container \"fd928d778461259ddc6b2ad5db79730cf4cac7d6f95c93d286982b58ef1acd7a\": container with ID starting with fd928d778461259ddc6b2ad5db79730cf4cac7d6f95c93d286982b58ef1acd7a not found: ID does not exist"
Sep 16 12:27:37.360906 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.360880    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"]
Sep 16 12:27:37.362661 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.362643    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef3317-6c54c4c5d-gzv7c"]
Sep 16 12:27:37.606479 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.606445    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:37.706859 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.706828    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:37.758103 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.758075    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="453f7dfa-4456-4a99-b2e3-6d4852d9d3ba" path="/var/lib/kubelet/pods/453f7dfa-4456-4a99-b2e3-6d4852d9d3ba/volumes"
Sep 16 12:27:37.942880 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:37.942774    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:38.733268 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:38.733227    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:39.411584 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:39.411529    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:40.046802 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:40.046763    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:40.134651 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:40.134620    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:40.843042 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:40.843008    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:42.272125 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:42.272090    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:43.951559 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.951510    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"]
Sep 16 12:27:43.951924 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.951908    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453f7dfa-4456-4a99-b2e3-6d4852d9d3ba" containerName="maas-api"
Sep 16 12:27:43.951924 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.951920    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="453f7dfa-4456-4a99-b2e3-6d4852d9d3ba" containerName="maas-api"
Sep 16 12:27:43.952007 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.951929    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb135408-2e81-4cca-9522-92d16cddb49d" containerName="payload-processing"
Sep 16 12:27:43.952007 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.951935    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb135408-2e81-4cca-9522-92d16cddb49d" containerName="payload-processing"
Sep 16 12:27:43.952007 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.951959    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b1f827-a3be-4282-b03a-2132d3b62edd" containerName="maas-api"
Sep 16 12:27:43.952007 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.951965    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b1f827-a3be-4282-b03a-2132d3b62edd" containerName="maas-api"
Sep 16 12:27:43.952126 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.952022    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb135408-2e81-4cca-9522-92d16cddb49d" containerName="payload-processing"
Sep 16 12:27:43.952126 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.952032    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="453f7dfa-4456-4a99-b2e3-6d4852d9d3ba" containerName="maas-api"
Sep 16 12:27:43.952126 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.952039    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b1f827-a3be-4282-b03a-2132d3b62edd" containerName="maas-api"
Sep 16 12:27:43.956391 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.956372    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:43.958856 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.958834    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"openshift-service-ca.crt\""
Sep 16 12:27:43.958986 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.958964    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"kube-root-ca.crt\""
Sep 16 12:27:43.959720 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.959702    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"e2e-conflict-b-786e64f4-kserve-self-signed-certs\""
Sep 16 12:27:43.959930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.959915    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"default-dockercfg-4xwdx\""
Sep 16 12:27:43.966614 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:43.966593    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"]
Sep 16 12:27:44.047440 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.047404    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:44.083351 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.083323    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-home\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.083504 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.083362    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d157c335-84df-4f78-a43e-c6ec6cc043a1-tls-certs\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.083504 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.083454    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wpzj\" (UniqueName: \"kubernetes.io/projected/d157c335-84df-4f78-a43e-c6ec6cc043a1-kube-api-access-6wpzj\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.083590 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.083510    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-dshm\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.083590 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.083538    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-model-cache\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.083658 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.083595    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-tmp-dir\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.184865 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.184831    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-model-cache\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.184865 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.184864    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-tmp-dir\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.185098 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.184893    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-home\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.185098 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.184920    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d157c335-84df-4f78-a43e-c6ec6cc043a1-tls-certs\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.185098 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.184979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wpzj\" (UniqueName: \"kubernetes.io/projected/d157c335-84df-4f78-a43e-c6ec6cc043a1-kube-api-access-6wpzj\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.185098 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.185045    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-dshm\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.185304 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.185259    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-model-cache\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.185304 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.185289    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-home\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.185412 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.185322    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-tmp-dir\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.187311 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.187288    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-dshm\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.187622 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.187600    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d157c335-84df-4f78-a43e-c6ec6cc043a1-tls-certs\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.191869 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.191850    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wpzj\" (UniqueName: \"kubernetes.io/projected/d157c335-84df-4f78-a43e-c6ec6cc043a1-kube-api-access-6wpzj\") pod \"e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.267059 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.266963    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:44.552670 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:44.552418    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"]
Sep 16 12:27:44.555456 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:27:44.555426    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd157c335_84df_4f78_a43e_c6ec6cc043a1.slice/crio-478c24be007025912d8016e95b97ff7825fef623d23f047a6d17198fd81f6a0a WatchSource:0}: Error finding container 478c24be007025912d8016e95b97ff7825fef623d23f047a6d17198fd81f6a0a: Status 404 returned error can't find the container with id 478c24be007025912d8016e95b97ff7825fef623d23f047a6d17198fd81f6a0a
Sep 16 12:27:45.381452 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:45.381414    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps" event={"ID":"d157c335-84df-4f78-a43e-c6ec6cc043a1","Type":"ContainerStarted","Data":"7a0d568a44f47be077021fc643bbf8e24769cd1527851426e017d06017ee1c37"}
Sep 16 12:27:45.381452 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:45.381456    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps" event={"ID":"d157c335-84df-4f78-a43e-c6ec6cc043a1","Type":"ContainerStarted","Data":"478c24be007025912d8016e95b97ff7825fef623d23f047a6d17198fd81f6a0a"}
Sep 16 12:27:45.397869 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:45.397819    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps" podStartSLOduration=2.397804566 podStartE2EDuration="2.397804566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:27:45.396143095 +0000 UTC m=+1462.150101825" watchObservedRunningTime="2026-09-16 12:27:45.397804566 +0000 UTC m=+1462.151763296"
Sep 16 12:27:52.465306 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:52.465273    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:54.267505 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:54.267472    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:54.267972 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:54.267532    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:54.279484 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:54.279459    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:27:54.437158 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:27:54.437123    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:28:02.620561 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.620508    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"]
Sep 16 12:28:02.624370 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.624349    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.626579 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.626532    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"distinct-2-w0-739ee7-kserve-self-signed-certs\""
Sep 16 12:28:02.630712 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.630691    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"]
Sep 16 12:28:02.645842 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.645808    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:02.755031 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.754999    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-dshm\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.755228 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.755054    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-home\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.755228 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.755159    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626a74de-6993-4a37-8f6e-207ec5d6190c-tls-certs\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.755228 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.755199    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-model-cache\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.755367 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.755256    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqw54\" (UniqueName: \"kubernetes.io/projected/626a74de-6993-4a37-8f6e-207ec5d6190c-kube-api-access-lqw54\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.755367 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.755311    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-tmp-dir\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.856046 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.856011    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-tmp-dir\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.856217 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.856061    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-dshm\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.856217 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.856106    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-home\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.856217 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.856139    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626a74de-6993-4a37-8f6e-207ec5d6190c-tls-certs\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.856217 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.856159    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-model-cache\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.856416 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.856386    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw54\" (UniqueName: \"kubernetes.io/projected/626a74de-6993-4a37-8f6e-207ec5d6190c-kube-api-access-lqw54\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.856473 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.856422    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-tmp-dir\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.856529 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.856477    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-model-cache\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.856617 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.856584    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-home\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.858477 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.858453    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-dshm\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.858673 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.858655    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626a74de-6993-4a37-8f6e-207ec5d6190c-tls-certs\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.863141 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.863119    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqw54\" (UniqueName: \"kubernetes.io/projected/626a74de-6993-4a37-8f6e-207ec5d6190c-kube-api-access-lqw54\") pod \"distinct-2-w0-739ee7-kserve-cdd98c568-mflk8\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") " pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:02.935502 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:02.935424    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:03.206972 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:03.206947    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"]
Sep 16 12:28:03.208921 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:28:03.208891    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod626a74de_6993_4a37_8f6e_207ec5d6190c.slice/crio-7e6f18942cd9d832ee4178e9f66bd7bd3883d05d163b0a4992336e67d9b854b6 WatchSource:0}: Error finding container 7e6f18942cd9d832ee4178e9f66bd7bd3883d05d163b0a4992336e67d9b854b6: Status 404 returned error can't find the container with id 7e6f18942cd9d832ee4178e9f66bd7bd3883d05d163b0a4992336e67d9b854b6
Sep 16 12:28:03.477420 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:03.477321    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8" event={"ID":"626a74de-6993-4a37-8f6e-207ec5d6190c","Type":"ContainerStarted","Data":"10af61903b84746fd6755c0d2956d332252cc6ac04dcd6d4ea442ab51fd39da2"}
Sep 16 12:28:03.477420 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:03.477369    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8" event={"ID":"626a74de-6993-4a37-8f6e-207ec5d6190c","Type":"ContainerStarted","Data":"7e6f18942cd9d832ee4178e9f66bd7bd3883d05d163b0a4992336e67d9b854b6"}
Sep 16 12:28:03.493578 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:03.493512    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8" podStartSLOduration=1.493498373 podStartE2EDuration="1.493498373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:28:03.492531959 +0000 UTC m=+1480.246490689" watchObservedRunningTime="2026-09-16 12:28:03.493498373 +0000 UTC m=+1480.247457104"
Sep 16 12:28:12.936093 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:12.936054    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:12.936093 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:12.936094    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:12.955075 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:12.955048    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:13.545085 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:13.545057    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:28:19.236226 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:19.236192    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:19.475183 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:19.475151    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:19.555957 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:19.555875    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:19.994898 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:19.994864    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:20.268356 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.268270    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"]
Sep 16 12:28:20.272138 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.272119    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.274821 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.274795    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-0a9008-openshift-default-dockercfg-b6m47\""
Sep 16 12:28:20.282091 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.282070    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"]
Sep 16 12:28:20.299223 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.299193    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:20.421400 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.421357    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-credential-socket\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.421605 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.421461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-data\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.421605 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.421491    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-token\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.421605 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.421524    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9rzc\" (UniqueName: \"kubernetes.io/projected/6c8318df-1a44-49c3-bc52-ceb2d94a2320-kube-api-access-x9rzc\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.421605 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.421577    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-workload-certs\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.421855 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.421610    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-podinfo\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.421855 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.421638    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-workload-socket\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.421855 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.421672    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-envoy\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.421855 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.421778    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istiod-ca-cert\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523144 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523046    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istiod-ca-cert\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523144 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523100    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-credential-socket\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523144 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523147    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-data\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523164    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-token\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523186    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9rzc\" (UniqueName: \"kubernetes.io/projected/6c8318df-1a44-49c3-bc52-ceb2d94a2320-kube-api-access-x9rzc\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523203    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-workload-certs\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523224    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-podinfo\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523251    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-workload-socket\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523283    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-envoy\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523764 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523656    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-credential-socket\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523764 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523669    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-data\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523871 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523841    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-workload-socket\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523917    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istiod-ca-cert\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.523991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.523932    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-workload-certs\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.525661 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.525635    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-envoy\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.525859 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.525840    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-podinfo\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.530145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.530123    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-token\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.530343 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.530327    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9rzc\" (UniqueName: \"kubernetes.io/projected/6c8318df-1a44-49c3-bc52-ceb2d94a2320-kube-api-access-x9rzc\") pod \"e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") " pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.586697 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.586669    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:20.753439 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.753407    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:20.859528 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.859502    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"]
Sep 16 12:28:20.861468 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:28:20.861436    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8318df_1a44_49c3_bc52_ceb2d94a2320.slice/crio-197e7ed6ccf231f01c07d5b3dac951d5508f1867291895f134613c162fb753a1 WatchSource:0}: Error finding container 197e7ed6ccf231f01c07d5b3dac951d5508f1867291895f134613c162fb753a1: Status 404 returned error can't find the container with id 197e7ed6ccf231f01c07d5b3dac951d5508f1867291895f134613c162fb753a1
Sep 16 12:28:20.864034 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.863987    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:28:20.864138 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.864088    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:28:20.864191 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:20.864134    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:28:21.581057 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:21.581024    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" event={"ID":"6c8318df-1a44-49c3-bc52-ceb2d94a2320","Type":"ContainerStarted","Data":"69806dd72ea2f753ca27fc451a416dfea49cd371e3a66fe9a08778177b5ca46a"}
Sep 16 12:28:21.581413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:21.581102    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" event={"ID":"6c8318df-1a44-49c3-bc52-ceb2d94a2320","Type":"ContainerStarted","Data":"197e7ed6ccf231f01c07d5b3dac951d5508f1867291895f134613c162fb753a1"}
Sep 16 12:28:21.587165 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:21.587143    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:21.591934 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:21.591910    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:21.601252 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:21.601204    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" podStartSLOduration=1.601187773 podStartE2EDuration="1.601187773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:28:21.600867375 +0000 UTC m=+1498.354826107" watchObservedRunningTime="2026-09-16 12:28:21.601187773 +0000 UTC m=+1498.355146506"
Sep 16 12:28:22.586598 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:22.586559    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:22.587600 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:22.587578    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:28:23.323610 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:23.323569    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:23.503209 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:23.503174    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:23.604677 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:23.604650    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:24.187797 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:24.187752    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:25.033322 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:25.033274    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:25.706204 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:25.706168    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:26.332853 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:26.332818    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:26.586221 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:26.586141    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:27.009133 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:27.009104    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:27.110087 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:27.110047    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:27.445265 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:27.445237    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:27.540075 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:27.540045    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:27.828973 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:27.828874    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:34.816909 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:34.816874    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:35.558769 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:35.558728    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:36.346770 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:36.346711    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:36.956773 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:36.956740    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:37.696772 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:37.696721    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:38.022344 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:38.022239    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"]
Sep 16 12:28:38.040403 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:38.040373    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:38.204602 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:38.204571    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:38.543091 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:38.543058    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:38.656305 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:38.656254    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:38.857193 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:38.857164    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:38.953677 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:38.953649    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:39.351593 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:39.351539    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:40.005600 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:40.005570    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:40.934031 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:40.933981    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:41.338132 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:41.338043    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:41.655199 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:41.655165    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:41.940291 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:41.940205    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:42.041590 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:42.041561    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:42.416761 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:42.416732    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:43.150508 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:43.150473    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:44.063092 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:44.063034    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:44.826192 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:44.826149    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:45.615760 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:45.615725    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:46.243875 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.243846    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:46.313523 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.313486    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:46.605950 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.605915    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:46.707925 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.707890    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:46.784265 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.784236    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:46.832308 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.832276    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"]
Sep 16 12:28:46.836033 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.836018    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"
Sep 16 12:28:46.838104 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.838080    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-0e94e5-dockercfg-xzqt5\""
Sep 16 12:28:46.838219 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.838093    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-0e94e5\""
Sep 16 12:28:46.846052 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.846026    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"]
Sep 16 12:28:46.867784 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.867716    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcb5s\" (UniqueName: \"kubernetes.io/projected/0b0a2c79-e014-4508-8d99-e3f51de91067-kube-api-access-bcb5s\") pod \"payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7\" (UID: \"0b0a2c79-e014-4508-8d99-e3f51de91067\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"
Sep 16 12:28:46.867784 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.867759    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0a2c79-e014-4508-8d99-e3f51de91067-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7\" (UID: \"0b0a2c79-e014-4508-8d99-e3f51de91067\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"
Sep 16 12:28:46.969230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.969194    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcb5s\" (UniqueName: \"kubernetes.io/projected/0b0a2c79-e014-4508-8d99-e3f51de91067-kube-api-access-bcb5s\") pod \"payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7\" (UID: \"0b0a2c79-e014-4508-8d99-e3f51de91067\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"
Sep 16 12:28:46.969427 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.969254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0a2c79-e014-4508-8d99-e3f51de91067-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7\" (UID: \"0b0a2c79-e014-4508-8d99-e3f51de91067\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"
Sep 16 12:28:46.970051 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.970025    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0a2c79-e014-4508-8d99-e3f51de91067-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7\" (UID: \"0b0a2c79-e014-4508-8d99-e3f51de91067\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"
Sep 16 12:28:46.977616 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:46.977592    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcb5s\" (UniqueName: \"kubernetes.io/projected/0b0a2c79-e014-4508-8d99-e3f51de91067-kube-api-access-bcb5s\") pod \"payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7\" (UID: \"0b0a2c79-e014-4508-8d99-e3f51de91067\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"
Sep 16 12:28:47.148284 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:47.148187    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"
Sep 16 12:28:47.420223 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:47.420196    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"]
Sep 16 12:28:47.422795 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:28:47.422757    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0a2c79_e014_4508_8d99_e3f51de91067.slice/crio-dbd65ae9b7838ef4b4d0346a233a5f607a6a648e09dcd42523a2c1938c2032fe WatchSource:0}: Error finding container dbd65ae9b7838ef4b4d0346a233a5f607a6a648e09dcd42523a2c1938c2032fe: Status 404 returned error can't find the container with id dbd65ae9b7838ef4b4d0346a233a5f607a6a648e09dcd42523a2c1938c2032fe
Sep 16 12:28:47.724615 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:47.724514    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7" event={"ID":"0b0a2c79-e014-4508-8d99-e3f51de91067","Type":"ContainerStarted","Data":"9cedb4dcda1b1585bcec89d339ee5530a3b78afe468fbf1e5957f5bbb9e63abf"}
Sep 16 12:28:47.724615 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:47.724583    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7" event={"ID":"0b0a2c79-e014-4508-8d99-e3f51de91067","Type":"ContainerStarted","Data":"dbd65ae9b7838ef4b4d0346a233a5f607a6a648e09dcd42523a2c1938c2032fe"}
Sep 16 12:28:47.724615 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:47.724614    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"
Sep 16 12:28:47.741252 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:47.741191    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7" podStartSLOduration=1.741175074 podStartE2EDuration="1.741175074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:28:47.739764675 +0000 UTC m=+1524.493723407" watchObservedRunningTime="2026-09-16 12:28:47.741175074 +0000 UTC m=+1524.495133804"
Sep 16 12:28:48.042191 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:48.042108    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:50.420234 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.420197    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:50.484697 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.484662    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"]
Sep 16 12:28:50.484931 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.484904    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht" podUID="acd69949-347b-40b7-819f-7a99e7042700" containerName="payload-processing" containerID="cri-o://3227fc57e9b5d1439a62cc834f03e6b3c0ab24ac718b8b6dad339f2f00abe8b2" gracePeriod=30
Sep 16 12:28:50.581086 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.581038    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:50.653302 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.653236    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:50.712912 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.712835    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:50.742953 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.742923    2558 generic.go:356] "Generic (PLEG): container finished" podID="acd69949-347b-40b7-819f-7a99e7042700" containerID="3227fc57e9b5d1439a62cc834f03e6b3c0ab24ac718b8b6dad339f2f00abe8b2" exitCode=0
Sep 16 12:28:50.743058 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.742983    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht" event={"ID":"acd69949-347b-40b7-819f-7a99e7042700","Type":"ContainerDied","Data":"3227fc57e9b5d1439a62cc834f03e6b3c0ab24ac718b8b6dad339f2f00abe8b2"}
Sep 16 12:28:50.743058 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.743036    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht" event={"ID":"acd69949-347b-40b7-819f-7a99e7042700","Type":"ContainerDied","Data":"3b1c87037ab2c21d4997df321faf72b8c64eaa811f5816bb145e1ad0a7530815"}
Sep 16 12:28:50.743131 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.743063    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b1c87037ab2c21d4997df321faf72b8c64eaa811f5816bb145e1ad0a7530815"
Sep 16 12:28:50.750291 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.750269    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"
Sep 16 12:28:50.809330 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.809300    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kpb8\" (UniqueName: \"kubernetes.io/projected/acd69949-347b-40b7-819f-7a99e7042700-kube-api-access-9kpb8\") pod \"acd69949-347b-40b7-819f-7a99e7042700\" (UID: \"acd69949-347b-40b7-819f-7a99e7042700\") "
Sep 16 12:28:50.809523 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.809380    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acd69949-347b-40b7-819f-7a99e7042700-plugins-config-volume\") pod \"acd69949-347b-40b7-819f-7a99e7042700\" (UID: \"acd69949-347b-40b7-819f-7a99e7042700\") "
Sep 16 12:28:50.809862 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.809829    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd69949-347b-40b7-819f-7a99e7042700-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "acd69949-347b-40b7-819f-7a99e7042700" (UID: "acd69949-347b-40b7-819f-7a99e7042700"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:50.811729 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.811700    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd69949-347b-40b7-819f-7a99e7042700-kube-api-access-9kpb8" (OuterVolumeSpecName: "kube-api-access-9kpb8") pod "acd69949-347b-40b7-819f-7a99e7042700" (UID: "acd69949-347b-40b7-819f-7a99e7042700"). InnerVolumeSpecName "kube-api-access-9kpb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:50.910883 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.910843    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acd69949-347b-40b7-819f-7a99e7042700-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:28:50.910883 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:50.910875    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kpb8\" (UniqueName: \"kubernetes.io/projected/acd69949-347b-40b7-819f-7a99e7042700-kube-api-access-9kpb8\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:28:51.160741 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:51.160685    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:51.734505 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:51.734475    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:51.749136 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:51.749103    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"
Sep 16 12:28:51.781942 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:51.781905    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"]
Sep 16 12:28:51.784274 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:51.784252    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5891124-6bbf985965-z2lht"]
Sep 16 12:28:51.802051 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:51.802026    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:51.952687 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:51.952645    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:52.623115 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:52.623071    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:53.042776 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.042688    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps" podUID="d157c335-84df-4f78-a43e-c6ec6cc043a1" containerName="main" containerID="cri-o://7a0d568a44f47be077021fc643bbf8e24769cd1527851426e017d06017ee1c37" gracePeriod=45
Sep 16 12:28:53.331779 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.331757    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:28:53.433858 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.433812    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:53.434817 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.434797    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-model-cache\") pod \"d157c335-84df-4f78-a43e-c6ec6cc043a1\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") "
Sep 16 12:28:53.434884 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.434874    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wpzj\" (UniqueName: \"kubernetes.io/projected/d157c335-84df-4f78-a43e-c6ec6cc043a1-kube-api-access-6wpzj\") pod \"d157c335-84df-4f78-a43e-c6ec6cc043a1\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") "
Sep 16 12:28:53.434929 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.434918    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-home\") pod \"d157c335-84df-4f78-a43e-c6ec6cc043a1\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") "
Sep 16 12:28:53.434966 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.434951    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-tmp-dir\") pod \"d157c335-84df-4f78-a43e-c6ec6cc043a1\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") "
Sep 16 12:28:53.435013 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.434975    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-dshm\") pod \"d157c335-84df-4f78-a43e-c6ec6cc043a1\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") "
Sep 16 12:28:53.435054 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.435010    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d157c335-84df-4f78-a43e-c6ec6cc043a1-tls-certs\") pod \"d157c335-84df-4f78-a43e-c6ec6cc043a1\" (UID: \"d157c335-84df-4f78-a43e-c6ec6cc043a1\") "
Sep 16 12:28:53.435102 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.435059    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "d157c335-84df-4f78-a43e-c6ec6cc043a1" (UID: "d157c335-84df-4f78-a43e-c6ec6cc043a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:53.435176 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.435155    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-home" (OuterVolumeSpecName: "home") pod "d157c335-84df-4f78-a43e-c6ec6cc043a1" (UID: "d157c335-84df-4f78-a43e-c6ec6cc043a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:53.435239 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.435168    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d157c335-84df-4f78-a43e-c6ec6cc043a1" (UID: "d157c335-84df-4f78-a43e-c6ec6cc043a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:53.435420 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.435399    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-home\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:28:53.435420 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.435419    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-tmp-dir\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:28:53.435677 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.435429    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-model-cache\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:28:53.437362 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.437309    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-dshm" (OuterVolumeSpecName: "dshm") pod "d157c335-84df-4f78-a43e-c6ec6cc043a1" (UID: "d157c335-84df-4f78-a43e-c6ec6cc043a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:53.437471 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.437422    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d157c335-84df-4f78-a43e-c6ec6cc043a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d157c335-84df-4f78-a43e-c6ec6cc043a1" (UID: "d157c335-84df-4f78-a43e-c6ec6cc043a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:53.437471 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.437437    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d157c335-84df-4f78-a43e-c6ec6cc043a1-kube-api-access-6wpzj" (OuterVolumeSpecName: "kube-api-access-6wpzj") pod "d157c335-84df-4f78-a43e-c6ec6cc043a1" (UID: "d157c335-84df-4f78-a43e-c6ec6cc043a1"). InnerVolumeSpecName "kube-api-access-6wpzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:53.536104 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.536073    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wpzj\" (UniqueName: \"kubernetes.io/projected/d157c335-84df-4f78-a43e-c6ec6cc043a1-kube-api-access-6wpzj\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:28:53.536104 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.536101    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d157c335-84df-4f78-a43e-c6ec6cc043a1-dshm\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:28:53.536104 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.536112    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d157c335-84df-4f78-a43e-c6ec6cc043a1-tls-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:28:53.766679 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.766636    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acd69949-347b-40b7-819f-7a99e7042700" path="/var/lib/kubelet/pods/acd69949-347b-40b7-819f-7a99e7042700/volumes"
Sep 16 12:28:53.776769 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.776733    2558 generic.go:356] "Generic (PLEG): container finished" podID="d157c335-84df-4f78-a43e-c6ec6cc043a1" containerID="7a0d568a44f47be077021fc643bbf8e24769cd1527851426e017d06017ee1c37" exitCode=0
Sep 16 12:28:53.776948 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.776786    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"
Sep 16 12:28:53.776948 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.776835    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps" event={"ID":"d157c335-84df-4f78-a43e-c6ec6cc043a1","Type":"ContainerDied","Data":"7a0d568a44f47be077021fc643bbf8e24769cd1527851426e017d06017ee1c37"}
Sep 16 12:28:53.776948 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.776877    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps" event={"ID":"d157c335-84df-4f78-a43e-c6ec6cc043a1","Type":"ContainerDied","Data":"478c24be007025912d8016e95b97ff7825fef623d23f047a6d17198fd81f6a0a"}
Sep 16 12:28:53.776948 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.776896    2558 scope.go:117] "RemoveContainer" containerID="7a0d568a44f47be077021fc643bbf8e24769cd1527851426e017d06017ee1c37"
Sep 16 12:28:53.787973 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.787950    2558 scope.go:117] "RemoveContainer" containerID="7a0d568a44f47be077021fc643bbf8e24769cd1527851426e017d06017ee1c37"
Sep 16 12:28:53.788266 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:28:53.788247    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0d568a44f47be077021fc643bbf8e24769cd1527851426e017d06017ee1c37\": container with ID starting with 7a0d568a44f47be077021fc643bbf8e24769cd1527851426e017d06017ee1c37 not found: ID does not exist" containerID="7a0d568a44f47be077021fc643bbf8e24769cd1527851426e017d06017ee1c37"
Sep 16 12:28:53.788311 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.788275    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0d568a44f47be077021fc643bbf8e24769cd1527851426e017d06017ee1c37"} err="failed to get container status \"7a0d568a44f47be077021fc643bbf8e24769cd1527851426e017d06017ee1c37\": rpc error: code = NotFound desc = could not find container \"7a0d568a44f47be077021fc643bbf8e24769cd1527851426e017d06017ee1c37\": container with ID starting with 7a0d568a44f47be077021fc643bbf8e24769cd1527851426e017d06017ee1c37 not found: ID does not exist"
Sep 16 12:28:53.796444 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.796416    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"]
Sep 16 12:28:53.799061 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.799040    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:53.799674 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:53.799655    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-b-786e64f4-kserve-5545499bf7-2lhps"]
Sep 16 12:28:54.261948 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:54.261917    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:55.119370 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:55.119332    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:55.763082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:55.762691    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d157c335-84df-4f78-a43e-c6ec6cc043a1" path="/var/lib/kubelet/pods/d157c335-84df-4f78-a43e-c6ec6cc043a1/volumes"
Sep 16 12:28:55.796971 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:55.796938    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:57.194204 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:57.194168    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:58.066890 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:58.066847    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:58.732328 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:58.732254    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"
Sep 16 12:28:58.751147 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:58.751115    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:58.887212 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:58.887184    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:59.481136 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:28:59.481099    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:29:00.238601 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:00.238533    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:29:00.868284 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:00.868248    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:29:01.602320 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:01.602272    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:29:01.739780 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:01.739630    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:29:01.822849 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:01.822816    2558 generic.go:356] "Generic (PLEG): container finished" podID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerID="69806dd72ea2f753ca27fc451a416dfea49cd371e3a66fe9a08778177b5ca46a" exitCode=0
Sep 16 12:29:01.823006 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:01.822891    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" event={"ID":"6c8318df-1a44-49c3-bc52-ceb2d94a2320","Type":"ContainerDied","Data":"69806dd72ea2f753ca27fc451a416dfea49cd371e3a66fe9a08778177b5ca46a"}
Sep 16 12:29:01.823312 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:01.823298    2558 scope.go:117] "RemoveContainer" containerID="69806dd72ea2f753ca27fc451a416dfea49cd371e3a66fe9a08778177b5ca46a"
Sep 16 12:29:01.824507 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:01.824479    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:29:01.824614 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:01.824560    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:29:01.824614 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:01.824606    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:29:02.829738 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:02.829698    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" event={"ID":"6c8318df-1a44-49c3-bc52-ceb2d94a2320","Type":"ContainerStarted","Data":"dd4473f1b437683373353ee3cb2c1d64338ddc4a6e87c42a3dbf67d809f5f00b"}
Sep 16 12:29:03.586858 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:03.586823    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:29:03.588160 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:03.588131    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:29:03.588289 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:03.588186    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 16 12:29:04.587330 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:04.587295    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:29:04.587734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:04.587352    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 16 12:29:05.587384 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:05.587347    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:29:05.587954 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:05.587411    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 16 12:29:06.587528 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:06.587489    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:29:06.587925 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:06.587561    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 16 12:29:07.230457 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.230419    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"]
Sep 16 12:29:07.231084 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.231062    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acd69949-347b-40b7-819f-7a99e7042700" containerName="payload-processing"
Sep 16 12:29:07.231084 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.231083    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="acd69949-347b-40b7-819f-7a99e7042700" containerName="payload-processing"
Sep 16 12:29:07.231232 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.231113    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d157c335-84df-4f78-a43e-c6ec6cc043a1" containerName="main"
Sep 16 12:29:07.231232 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.231162    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d157c335-84df-4f78-a43e-c6ec6cc043a1" containerName="main"
Sep 16 12:29:07.231315 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.231265    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d157c335-84df-4f78-a43e-c6ec6cc043a1" containerName="main"
Sep 16 12:29:07.231315 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.231276    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="acd69949-347b-40b7-819f-7a99e7042700" containerName="payload-processing"
Sep 16 12:29:07.234814 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.234791    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.236850 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.236829    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"default-dockercfg-4xwdx\""
Sep 16 12:29:07.236997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.236977    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"openshift-service-ca.crt\""
Sep 16 12:29:07.237788 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.237766    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"kube-root-ca.crt\""
Sep 16 12:29:07.237788 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.237781    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"distinct-w4-68f7cd-kserve-self-signed-certs\""
Sep 16 12:29:07.241911 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.241890    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"]
Sep 16 12:29:07.271993 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.271960    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:29:07.390953 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.390918    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-home\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.390953 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.390958    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-model-cache\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.391177 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.390986    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-tls-certs\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.391177 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.391047    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-dshm\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.391177 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.391071    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-tmp-dir\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.391177 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.391109    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw2pt\" (UniqueName: \"kubernetes.io/projected/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-kube-api-access-hw2pt\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.491932 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.491847    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-home\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.491932 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.491889    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-model-cache\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.491932 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.491915    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-tls-certs\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.491932 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.491932    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-dshm\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.492267 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.491951    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-tmp-dir\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.492267 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.491994    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw2pt\" (UniqueName: \"kubernetes.io/projected/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-kube-api-access-hw2pt\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.492380 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.492341    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-model-cache\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.492380 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.492341    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-home\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.492469 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.492428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-tmp-dir\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.494290 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.494266    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-dshm\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.494399 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.494380    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-tls-certs\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.498861 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.498841    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw2pt\" (UniqueName: \"kubernetes.io/projected/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-kube-api-access-hw2pt\") pod \"distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.545853 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.545805    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:07.589578 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.586648    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:29:07.589578 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.587782    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:29:07.589578 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.587832    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 16 12:29:07.844556 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.844517    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"]
Sep 16 12:29:07.847610 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:29:07.847538    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bf41d9b_0610_430f_86fd_52c6c8ff0d35.slice/crio-4f772344da8c5366fa3b7286f93840ed93f38793a46f3109d1ffb347d55d83ea WatchSource:0}: Error finding container 4f772344da8c5366fa3b7286f93840ed93f38793a46f3109d1ffb347d55d83ea: Status 404 returned error can't find the container with id 4f772344da8c5366fa3b7286f93840ed93f38793a46f3109d1ffb347d55d83ea
Sep 16 12:29:07.877159 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:07.877129    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz" event={"ID":"1bf41d9b-0610-430f-86fd-52c6c8ff0d35","Type":"ContainerStarted","Data":"4f772344da8c5366fa3b7286f93840ed93f38793a46f3109d1ffb347d55d83ea"}
Sep 16 12:29:08.591160 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:08.591130    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:29:08.592064 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:08.592044    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:29:08.883393 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:08.883295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz" event={"ID":"1bf41d9b-0610-430f-86fd-52c6c8ff0d35","Type":"ContainerStarted","Data":"19e08067be0596847dc96dc8c1f3ac021a714a7631863ff0b9fd87aa4c99b842"}
Sep 16 12:29:08.900839 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:08.900793    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz" podStartSLOduration=1.900775496 podStartE2EDuration="1.900775496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:29:08.898596357 +0000 UTC m=+1545.652555088" watchObservedRunningTime="2026-09-16 12:29:08.900775496 +0000 UTC m=+1545.654734226"
Sep 16 12:29:17.546671 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:17.546619    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:17.547100 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:17.546906    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:17.559935 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:17.559907    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:17.942869 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:17.942832    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:29:35.372329 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:35.372296    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:29:35.820945 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:35.820849    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:29:41.330072 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:41.330034    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:29:45.274083 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:45.274052    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:29:46.644772 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:29:46.644722    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:00.146332 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:00.146298    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c"]
Sep 16 12:30:00.150005 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:00.149980    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c"
Sep 16 12:30:00.152447 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:00.152427    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:30:00.158950 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:00.158921    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c"]
Sep 16 12:30:00.163104 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:00.163074    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c"
Sep 16 12:30:00.437885 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:00.437854    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c"]
Sep 16 12:30:00.440299 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:30:00.440272    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9548ccb9_050b_46db_8d90_0b0da20f932a.slice/crio-e0c8e0b8203a3ed6a11f7222bfd377dcd85d3119ea80d349edd2584867130aa0 WatchSource:0}: Error finding container e0c8e0b8203a3ed6a11f7222bfd377dcd85d3119ea80d349edd2584867130aa0: Status 404 returned error can't find the container with id e0c8e0b8203a3ed6a11f7222bfd377dcd85d3119ea80d349edd2584867130aa0
Sep 16 12:30:00.442239 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:00.442220    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:30:01.163200 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:01.163164    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c" event={"ID":"9548ccb9-050b-46db-8d90-0b0da20f932a","Type":"ContainerStarted","Data":"e0c8e0b8203a3ed6a11f7222bfd377dcd85d3119ea80d349edd2584867130aa0"}
Sep 16 12:30:03.177677 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:03.177640    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c" event={"ID":"9548ccb9-050b-46db-8d90-0b0da20f932a","Type":"ContainerStarted","Data":"cca1792f4c667a65ba919f1a788152cff1c694ace42b567c8ae750424ad8f0d6"}
Sep 16 12:30:03.200878 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:03.200822    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c" podStartSLOduration=0.805351541 podStartE2EDuration="3.200806331s" totalImagesPullingTime="2.39545479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:30:03.198271548 +0000 UTC m=+1599.952230279" watchObservedRunningTime="2026-09-16 12:30:03.200806331 +0000 UTC m=+1599.954765061"
Sep 16 12:30:04.185265 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:04.185176    2558 generic.go:356] "Generic (PLEG): container finished" podID="9548ccb9-050b-46db-8d90-0b0da20f932a" containerID="cca1792f4c667a65ba919f1a788152cff1c694ace42b567c8ae750424ad8f0d6" exitCode=0
Sep 16 12:30:04.185643 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:04.185270    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c" event={"ID":"9548ccb9-050b-46db-8d90-0b0da20f932a","Type":"ContainerDied","Data":"cca1792f4c667a65ba919f1a788152cff1c694ace42b567c8ae750424ad8f0d6"}
Sep 16 12:30:04.920843 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:04.920802    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:05.351810 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:05.351778    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c"
Sep 16 12:30:06.198386 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:06.198354    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c"
Sep 16 12:30:06.198631 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:06.198386    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c" event={"ID":"9548ccb9-050b-46db-8d90-0b0da20f932a","Type":"ContainerDied","Data":"e0c8e0b8203a3ed6a11f7222bfd377dcd85d3119ea80d349edd2584867130aa0"}
Sep 16 12:30:06.198631 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:06.198436    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0c8e0b8203a3ed6a11f7222bfd377dcd85d3119ea80d349edd2584867130aa0"
Sep 16 12:30:07.361445 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:07.361409    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:07.462257 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:07.462226    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:08.922067 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:08.922030    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:11.071918 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:11.071870    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:14.971509 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:14.971466    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:15.050470 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:15.050436    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:15.634291 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:15.634257    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:16.064309 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:16.064204    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:21.640491 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:21.640450    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:21.713697 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:21.713657    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:21.840147 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:21.840113    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:23.541451 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.541419    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"]
Sep 16 12:30:23.541885 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.541871    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9548ccb9-050b-46db-8d90-0b0da20f932a" containerName="cleanup"
Sep 16 12:30:23.541930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.541886    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9548ccb9-050b-46db-8d90-0b0da20f932a" containerName="cleanup"
Sep 16 12:30:23.541983 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.541974    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9548ccb9-050b-46db-8d90-0b0da20f932a" containerName="cleanup"
Sep 16 12:30:23.545125 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.545110    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.548131 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.548104    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"default-dockercfg-vbx8f\""
Sep 16 12:30:23.548263 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.548154    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"kube-root-ca.crt\""
Sep 16 12:30:23.548263 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.548175    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"auth-test-model-0e94e5-kserve-self-signed-certs\""
Sep 16 12:30:23.548263 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.548154    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"openshift-service-ca.crt\""
Sep 16 12:30:23.552441 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.552420    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"]
Sep 16 12:30:23.572075 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.572048    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-tmp-dir\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.572241 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.572113    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f906602e-dd9d-47d5-b03f-d44c09960bc9-tls-certs\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.572306 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.572284    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-model-cache\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.572349 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.572337    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-dshm\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.572403 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.572391    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-home\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.572441 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.572415    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b8d2\" (UniqueName: \"kubernetes.io/projected/f906602e-dd9d-47d5-b03f-d44c09960bc9-kube-api-access-5b8d2\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.657127 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.657099    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:23.673357 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.673334    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-model-cache\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.673488 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.673378    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-dshm\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.673488 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.673404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-home\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.673630 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.673531    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b8d2\" (UniqueName: \"kubernetes.io/projected/f906602e-dd9d-47d5-b03f-d44c09960bc9-kube-api-access-5b8d2\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.673695 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.673651    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-tmp-dir\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.673695 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.673682    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f906602e-dd9d-47d5-b03f-d44c09960bc9-tls-certs\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.673795 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.673773    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-home\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.673879 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.673853    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-model-cache\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.674019 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.673995    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-tmp-dir\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.675652 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.675627    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-dshm\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.675842 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.675824    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"auth-test-model-0e94e5-kserve-self-signed-certs\""
Sep 16 12:30:23.682749 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.682717    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"kube-root-ca.crt\""
Sep 16 12:30:23.686266 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.686246    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f906602e-dd9d-47d5-b03f-d44c09960bc9-tls-certs\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.693176 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.693155    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"openshift-service-ca.crt\""
Sep 16 12:30:23.704189 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.704162    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b8d2\" (UniqueName: \"kubernetes.io/projected/f906602e-dd9d-47d5-b03f-d44c09960bc9-kube-api-access-5b8d2\") pod \"auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:23.859958 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.859931    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"default-dockercfg-vbx8f\""
Sep 16 12:30:23.868263 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:23.868241    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:24.148669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:24.145737    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"]
Sep 16 12:30:24.149269 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:30:24.149244    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf906602e_dd9d_47d5_b03f_d44c09960bc9.slice/crio-a25b986e411b526718ca6738376651f07433f8a550fc50aeb28f12eadd1a4557 WatchSource:0}: Error finding container a25b986e411b526718ca6738376651f07433f8a550fc50aeb28f12eadd1a4557: Status 404 returned error can't find the container with id a25b986e411b526718ca6738376651f07433f8a550fc50aeb28f12eadd1a4557
Sep 16 12:30:24.297344 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:24.297270    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg" event={"ID":"f906602e-dd9d-47d5-b03f-d44c09960bc9","Type":"ContainerStarted","Data":"49d2c373192974af70d2367e698c269c0e3b1ad06816df325c54b35b7f4b6585"}
Sep 16 12:30:24.297344 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:24.297315    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg" event={"ID":"f906602e-dd9d-47d5-b03f-d44c09960bc9","Type":"ContainerStarted","Data":"a25b986e411b526718ca6738376651f07433f8a550fc50aeb28f12eadd1a4557"}
Sep 16 12:30:24.316624 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:24.316575    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg" podStartSLOduration=1.316560004 podStartE2EDuration="1.316560004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:30:24.315229816 +0000 UTC m=+1621.069188559" watchObservedRunningTime="2026-09-16 12:30:24.316560004 +0000 UTC m=+1621.070518730"
Sep 16 12:30:30.717741 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:30.717708    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:32.341037 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:32.341004    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:32.563677 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:32.563643    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:33.869067 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:33.869022    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:33.869067 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:33.869073    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:33.881909 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:33.881878    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:34.368454 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:34.368425    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:30:36.078703 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:36.078653    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:38.065280 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:38.065239    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:53.515682 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:53.515645    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:58.141315 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:30:58.141271    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:00.785326 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:00.785289    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:00.895684 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:00.895651    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:01.788395 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:01.788353    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:01.886141 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:01.886103    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:01.991487 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:01.991451    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:03.121297 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:03.121262    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:04.644751 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:04.644705    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:05.335215 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:05.335183    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:05.478403 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:05.478364    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:05.934375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:05.934347    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:07.133755 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:07.133719    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:08.436503 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:08.436463    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:09.898447 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:09.898414    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:11.726399 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:11.726361    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:12.870178 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:12.870140    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:14.312032 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:14.311992    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:14.515120 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:14.515084    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:14.846233 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:14.846202    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:15.435238 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:15.435201    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:15.542116 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:15.542066    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:15.679110 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:15.679074    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"]
Sep 16 12:31:15.689905 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:15.684854    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"
Sep 16 12:31:15.701112 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:15.701080    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8641cd\""
Sep 16 12:31:15.702111 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:15.702084    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8641cd-dockercfg-g56mz\""
Sep 16 12:31:15.705295 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:15.705265    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"]
Sep 16 12:31:15.820296 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:15.820261    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b482b9a7-951c-4a0c-bc34-90aa0c70c632-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv\" (UID: \"b482b9a7-951c-4a0c-bc34-90aa0c70c632\") " pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"
Sep 16 12:31:15.820496 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:15.820416    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfkcf\" (UniqueName: \"kubernetes.io/projected/b482b9a7-951c-4a0c-bc34-90aa0c70c632-kube-api-access-tfkcf\") pod \"payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv\" (UID: \"b482b9a7-951c-4a0c-bc34-90aa0c70c632\") " pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"
Sep 16 12:31:15.921923 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:15.921885    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfkcf\" (UniqueName: \"kubernetes.io/projected/b482b9a7-951c-4a0c-bc34-90aa0c70c632-kube-api-access-tfkcf\") pod \"payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv\" (UID: \"b482b9a7-951c-4a0c-bc34-90aa0c70c632\") " pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"
Sep 16 12:31:15.922156 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:15.922044    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b482b9a7-951c-4a0c-bc34-90aa0c70c632-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv\" (UID: \"b482b9a7-951c-4a0c-bc34-90aa0c70c632\") " pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"
Sep 16 12:31:15.922757 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:15.922735    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b482b9a7-951c-4a0c-bc34-90aa0c70c632-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv\" (UID: \"b482b9a7-951c-4a0c-bc34-90aa0c70c632\") " pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"
Sep 16 12:31:15.930754 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:15.930723    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfkcf\" (UniqueName: \"kubernetes.io/projected/b482b9a7-951c-4a0c-bc34-90aa0c70c632-kube-api-access-tfkcf\") pod \"payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv\" (UID: \"b482b9a7-951c-4a0c-bc34-90aa0c70c632\") " pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"
Sep 16 12:31:16.005620 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:16.005521    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"
Sep 16 12:31:16.307662 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:31:16.307632    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb482b9a7_951c_4a0c_bc34_90aa0c70c632.slice/crio-be1f5330ba79bbfc9e876d80b1a9276642fbdd45cca1a6e540b778a47448cbf9 WatchSource:0}: Error finding container be1f5330ba79bbfc9e876d80b1a9276642fbdd45cca1a6e540b778a47448cbf9: Status 404 returned error can't find the container with id be1f5330ba79bbfc9e876d80b1a9276642fbdd45cca1a6e540b778a47448cbf9
Sep 16 12:31:16.318982 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:16.318957    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"]
Sep 16 12:31:16.587283 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:16.587186    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv" event={"ID":"b482b9a7-951c-4a0c-bc34-90aa0c70c632","Type":"ContainerStarted","Data":"9bff68982d0bcf782f9b2296c2a67ebb4edae89ecb24f26877f58ea1a755e9d0"}
Sep 16 12:31:16.587283 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:16.587245    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv" event={"ID":"b482b9a7-951c-4a0c-bc34-90aa0c70c632","Type":"ContainerStarted","Data":"be1f5330ba79bbfc9e876d80b1a9276642fbdd45cca1a6e540b778a47448cbf9"}
Sep 16 12:31:16.587822 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:16.587318    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"
Sep 16 12:31:16.619493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:16.619437    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv" podStartSLOduration=1.619419555 podStartE2EDuration="1.619419555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:31:16.616452366 +0000 UTC m=+1673.370411098" watchObservedRunningTime="2026-09-16 12:31:16.619419555 +0000 UTC m=+1673.373378285"
Sep 16 12:31:27.596355 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:27.596322    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"
Sep 16 12:31:36.880889 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:36.880848    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:37.971760 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:37.971726    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:38.630770 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:38.630733    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:39.258187 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:39.258144    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:40.416083 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:40.416046    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:40.602198 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:40.602162    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:40.980703 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:40.980671    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:41.563803 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.563754    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"]
Sep 16 12:31:41.567610 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.567588    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.572044 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.572021    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-352217-openshift-default-dockercfg-9jp86\""
Sep 16 12:31:41.593257 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.593223    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:41.603515 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.603480    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"]
Sep 16 12:31:41.683948 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.683910    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phd2r\" (UniqueName: \"kubernetes.io/projected/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-kube-api-access-phd2r\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.684147 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.683967    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-envoy\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.684147 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.684016    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-podinfo\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.684147 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.684054    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istiod-ca-cert\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.684147 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.684089    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-workload-socket\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.684147 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.684132    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-data\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.684370 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.684192    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-workload-certs\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.684370 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.684236    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-token\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.684370 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.684302    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-credential-socket\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.785229 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.785187    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-workload-certs\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.785229 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.785226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-token\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.785493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.785268    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-credential-socket\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.785493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.785291    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phd2r\" (UniqueName: \"kubernetes.io/projected/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-kube-api-access-phd2r\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.785493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.785332    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-envoy\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.785493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.785369    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-podinfo\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.785493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.785412    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istiod-ca-cert\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.785493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.785441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-workload-socket\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.785493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.785475    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-data\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.786069 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.786045    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-workload-certs\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.786166 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.786079    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-workload-socket\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.786166 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.786149    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-credential-socket\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.786634 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.786541    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istiod-ca-cert\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.786759 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.786741    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-data\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.788110 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.788087    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-envoy\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.788225 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.788203    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-podinfo\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.807605 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.807573    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phd2r\" (UniqueName: \"kubernetes.io/projected/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-kube-api-access-phd2r\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.808149 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.808126    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-token\") pod \"e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") " pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.881447 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.881405    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:41.923250 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:41.923215    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:42.206154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:42.206127    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"]
Sep 16 12:31:42.208387 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:31:42.208358    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7568bd_7ee9_419c_b2eb_d4ce3df8e6da.slice/crio-d9ceec1617fc86ea6e23f880a3186dcca2da8a5e6688cc6e8d4aed308c5cda8a WatchSource:0}: Error finding container d9ceec1617fc86ea6e23f880a3186dcca2da8a5e6688cc6e8d4aed308c5cda8a: Status 404 returned error can't find the container with id d9ceec1617fc86ea6e23f880a3186dcca2da8a5e6688cc6e8d4aed308c5cda8a
Sep 16 12:31:42.210749 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:42.210716    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:31:42.210815 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:42.210786    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:31:42.210852 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:42.210838    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:31:42.285676 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:42.285638    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:42.740377 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:42.740345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz" event={"ID":"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da","Type":"ContainerStarted","Data":"12bc4f4525bc1aa08aac6b244682db3764fa840736ca5a14c2524fbb47bf0440"}
Sep 16 12:31:42.740794 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:42.740397    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz" event={"ID":"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da","Type":"ContainerStarted","Data":"d9ceec1617fc86ea6e23f880a3186dcca2da8a5e6688cc6e8d4aed308c5cda8a"}
Sep 16 12:31:42.881886 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:42.881837    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:43.743349 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:43.743294    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz" podStartSLOduration=2.743274784 podStartE2EDuration="2.743274784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:31:42.801086975 +0000 UTC m=+1699.555045706" watchObservedRunningTime="2026-09-16 12:31:43.743274784 +0000 UTC m=+1700.497233517"
Sep 16 12:31:43.744096 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:43.743490    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"]
Sep 16 12:31:43.761049 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:43.761020    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"]
Sep 16 12:31:43.888214 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:43.888179    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:44.099809 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:44.099770    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:44.702823 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:44.702784    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:44.752768 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:44.752733    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:44.753903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:44.753880    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:31:45.351649 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:45.351613    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:45.547591 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:45.547531    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:46.586074 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:46.586001    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:48.371752 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:48.371724    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:50.129728 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:50.129682    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:52.170052 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:52.170019    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:54.081816 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:54.081777    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:54.229713 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:54.229677    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:54.391136 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:54.391100    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:55.096644 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:55.096607    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:55.328093 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:55.328060    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:56.048681 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:56.048649    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:57.664069 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:57.664019    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:58.765065 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:58.765004    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8" podUID="626a74de-6993-4a37-8f6e-207ec5d6190c" containerName="main" containerID="cri-o://10af61903b84746fd6755c0d2956d332252cc6ac04dcd6d4ea442ab51fd39da2" gracePeriod=45
Sep 16 12:31:58.779921 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:58.779807    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957" podUID="609f3195-96fa-45aa-842a-80222b4bf8ee" containerName="main" containerID="cri-o://7255c788ace52c41427445001fb35a3ed1ba01a99601e71a5bcbb62d3356e507" gracePeriod=45
Sep 16 12:31:59.061621 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.061583    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:59.063024 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.063004    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:31:59.086507 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.086478    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:31:59.173347 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.173313    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-home\") pod \"609f3195-96fa-45aa-842a-80222b4bf8ee\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") "
Sep 16 12:31:59.173533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.173380    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626a74de-6993-4a37-8f6e-207ec5d6190c-tls-certs\") pod \"626a74de-6993-4a37-8f6e-207ec5d6190c\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") "
Sep 16 12:31:59.173533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.173426    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqw54\" (UniqueName: \"kubernetes.io/projected/626a74de-6993-4a37-8f6e-207ec5d6190c-kube-api-access-lqw54\") pod \"626a74de-6993-4a37-8f6e-207ec5d6190c\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") "
Sep 16 12:31:59.173533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.173461    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609f3195-96fa-45aa-842a-80222b4bf8ee-tls-certs\") pod \"609f3195-96fa-45aa-842a-80222b4bf8ee\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") "
Sep 16 12:31:59.173533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.173486    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-tmp-dir\") pod \"626a74de-6993-4a37-8f6e-207ec5d6190c\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") "
Sep 16 12:31:59.173798 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.173747    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "626a74de-6993-4a37-8f6e-207ec5d6190c" (UID: "626a74de-6993-4a37-8f6e-207ec5d6190c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.173798 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.173747    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-home" (OuterVolumeSpecName: "home") pod "609f3195-96fa-45aa-842a-80222b4bf8ee" (UID: "609f3195-96fa-45aa-842a-80222b4bf8ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.174020 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.173995    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-home\") pod \"626a74de-6993-4a37-8f6e-207ec5d6190c\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") "
Sep 16 12:31:59.174126 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.174091    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-dshm\") pod \"626a74de-6993-4a37-8f6e-207ec5d6190c\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") "
Sep 16 12:31:59.174126 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.174120    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-dshm\") pod \"609f3195-96fa-45aa-842a-80222b4bf8ee\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") "
Sep 16 12:31:59.174246 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.174153    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-tmp-dir\") pod \"609f3195-96fa-45aa-842a-80222b4bf8ee\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") "
Sep 16 12:31:59.174246 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.174179    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-model-cache\") pod \"626a74de-6993-4a37-8f6e-207ec5d6190c\" (UID: \"626a74de-6993-4a37-8f6e-207ec5d6190c\") "
Sep 16 12:31:59.174246 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.174211    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-model-cache\") pod \"609f3195-96fa-45aa-842a-80222b4bf8ee\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") "
Sep 16 12:31:59.174413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.174263    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6x8c\" (UniqueName: \"kubernetes.io/projected/609f3195-96fa-45aa-842a-80222b4bf8ee-kube-api-access-r6x8c\") pod \"609f3195-96fa-45aa-842a-80222b4bf8ee\" (UID: \"609f3195-96fa-45aa-842a-80222b4bf8ee\") "
Sep 16 12:31:59.174846 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.174808    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-home\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.174846 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.174796    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-model-cache" (OuterVolumeSpecName: "model-cache") pod "626a74de-6993-4a37-8f6e-207ec5d6190c" (UID: "626a74de-6993-4a37-8f6e-207ec5d6190c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.174998 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.174857    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-tmp-dir\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.175057 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.175014    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "609f3195-96fa-45aa-842a-80222b4bf8ee" (UID: "609f3195-96fa-45aa-842a-80222b4bf8ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.175387 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.175163    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-home" (OuterVolumeSpecName: "home") pod "626a74de-6993-4a37-8f6e-207ec5d6190c" (UID: "626a74de-6993-4a37-8f6e-207ec5d6190c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.175582 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.175510    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-model-cache" (OuterVolumeSpecName: "model-cache") pod "609f3195-96fa-45aa-842a-80222b4bf8ee" (UID: "609f3195-96fa-45aa-842a-80222b4bf8ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.176596 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.176570    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/609f3195-96fa-45aa-842a-80222b4bf8ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "609f3195-96fa-45aa-842a-80222b4bf8ee" (UID: "609f3195-96fa-45aa-842a-80222b4bf8ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:31:59.176702 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.176618    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626a74de-6993-4a37-8f6e-207ec5d6190c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "626a74de-6993-4a37-8f6e-207ec5d6190c" (UID: "626a74de-6993-4a37-8f6e-207ec5d6190c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:31:59.177028 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.177001    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626a74de-6993-4a37-8f6e-207ec5d6190c-kube-api-access-lqw54" (OuterVolumeSpecName: "kube-api-access-lqw54") pod "626a74de-6993-4a37-8f6e-207ec5d6190c" (UID: "626a74de-6993-4a37-8f6e-207ec5d6190c"). InnerVolumeSpecName "kube-api-access-lqw54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:31:59.177192 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.177169    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609f3195-96fa-45aa-842a-80222b4bf8ee-kube-api-access-r6x8c" (OuterVolumeSpecName: "kube-api-access-r6x8c") pod "609f3195-96fa-45aa-842a-80222b4bf8ee" (UID: "609f3195-96fa-45aa-842a-80222b4bf8ee"). InnerVolumeSpecName "kube-api-access-r6x8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:31:59.178362 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.178316    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-dshm" (OuterVolumeSpecName: "dshm") pod "609f3195-96fa-45aa-842a-80222b4bf8ee" (UID: "609f3195-96fa-45aa-842a-80222b4bf8ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.178464 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.178405    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-dshm" (OuterVolumeSpecName: "dshm") pod "626a74de-6993-4a37-8f6e-207ec5d6190c" (UID: "626a74de-6993-4a37-8f6e-207ec5d6190c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.275422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.275384    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-home\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.275422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.275416    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-dshm\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.275422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.275426    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-dshm\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.275707 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.275437    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-tmp-dir\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.275707 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.275450    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626a74de-6993-4a37-8f6e-207ec5d6190c-model-cache\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.275707 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.275461    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609f3195-96fa-45aa-842a-80222b4bf8ee-model-cache\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.275707 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.275473    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6x8c\" (UniqueName: \"kubernetes.io/projected/609f3195-96fa-45aa-842a-80222b4bf8ee-kube-api-access-r6x8c\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.275707 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.275483    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626a74de-6993-4a37-8f6e-207ec5d6190c-tls-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.275707 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.275495    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqw54\" (UniqueName: \"kubernetes.io/projected/626a74de-6993-4a37-8f6e-207ec5d6190c-kube-api-access-lqw54\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.275707 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.275508    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609f3195-96fa-45aa-842a-80222b4bf8ee-tls-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.843138 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.843103    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957" event={"ID":"609f3195-96fa-45aa-842a-80222b4bf8ee","Type":"ContainerDied","Data":"7255c788ace52c41427445001fb35a3ed1ba01a99601e71a5bcbb62d3356e507"}
Sep 16 12:31:59.843138 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.843139    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"
Sep 16 12:31:59.843667 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.843159    2558 scope.go:117] "RemoveContainer" containerID="7255c788ace52c41427445001fb35a3ed1ba01a99601e71a5bcbb62d3356e507"
Sep 16 12:31:59.843667 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.843106    2558 generic.go:356] "Generic (PLEG): container finished" podID="609f3195-96fa-45aa-842a-80222b4bf8ee" containerID="7255c788ace52c41427445001fb35a3ed1ba01a99601e71a5bcbb62d3356e507" exitCode=0
Sep 16 12:31:59.843667 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.843288    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957" event={"ID":"609f3195-96fa-45aa-842a-80222b4bf8ee","Type":"ContainerDied","Data":"766ebc018b04ca03a70ef1e1266eba5696436c1c528d0d1279b68ecbdd537902"}
Sep 16 12:31:59.845142 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.845116    2558 generic.go:356] "Generic (PLEG): container finished" podID="626a74de-6993-4a37-8f6e-207ec5d6190c" containerID="10af61903b84746fd6755c0d2956d332252cc6ac04dcd6d4ea442ab51fd39da2" exitCode=0
Sep 16 12:31:59.845265 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.845175    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"
Sep 16 12:31:59.845265 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.845203    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8" event={"ID":"626a74de-6993-4a37-8f6e-207ec5d6190c","Type":"ContainerDied","Data":"10af61903b84746fd6755c0d2956d332252cc6ac04dcd6d4ea442ab51fd39da2"}
Sep 16 12:31:59.845265 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.845246    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8" event={"ID":"626a74de-6993-4a37-8f6e-207ec5d6190c","Type":"ContainerDied","Data":"7e6f18942cd9d832ee4178e9f66bd7bd3883d05d163b0a4992336e67d9b854b6"}
Sep 16 12:31:59.852980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.852955    2558 scope.go:117] "RemoveContainer" containerID="7255c788ace52c41427445001fb35a3ed1ba01a99601e71a5bcbb62d3356e507"
Sep 16 12:31:59.853287 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:31:59.853268    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7255c788ace52c41427445001fb35a3ed1ba01a99601e71a5bcbb62d3356e507\": container with ID starting with 7255c788ace52c41427445001fb35a3ed1ba01a99601e71a5bcbb62d3356e507 not found: ID does not exist" containerID="7255c788ace52c41427445001fb35a3ed1ba01a99601e71a5bcbb62d3356e507"
Sep 16 12:31:59.853351 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.853295    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7255c788ace52c41427445001fb35a3ed1ba01a99601e71a5bcbb62d3356e507"} err="failed to get container status \"7255c788ace52c41427445001fb35a3ed1ba01a99601e71a5bcbb62d3356e507\": rpc error: code = NotFound desc = could not find container \"7255c788ace52c41427445001fb35a3ed1ba01a99601e71a5bcbb62d3356e507\": container with ID starting with 7255c788ace52c41427445001fb35a3ed1ba01a99601e71a5bcbb62d3356e507 not found: ID does not exist"
Sep 16 12:31:59.853351 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.853313    2558 scope.go:117] "RemoveContainer" containerID="10af61903b84746fd6755c0d2956d332252cc6ac04dcd6d4ea442ab51fd39da2"
Sep 16 12:31:59.862024 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.861999    2558 scope.go:117] "RemoveContainer" containerID="10af61903b84746fd6755c0d2956d332252cc6ac04dcd6d4ea442ab51fd39da2"
Sep 16 12:31:59.862323 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:31:59.862298    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10af61903b84746fd6755c0d2956d332252cc6ac04dcd6d4ea442ab51fd39da2\": container with ID starting with 10af61903b84746fd6755c0d2956d332252cc6ac04dcd6d4ea442ab51fd39da2 not found: ID does not exist" containerID="10af61903b84746fd6755c0d2956d332252cc6ac04dcd6d4ea442ab51fd39da2"
Sep 16 12:31:59.862382 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.862335    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10af61903b84746fd6755c0d2956d332252cc6ac04dcd6d4ea442ab51fd39da2"} err="failed to get container status \"10af61903b84746fd6755c0d2956d332252cc6ac04dcd6d4ea442ab51fd39da2\": rpc error: code = NotFound desc = could not find container \"10af61903b84746fd6755c0d2956d332252cc6ac04dcd6d4ea442ab51fd39da2\": container with ID starting with 10af61903b84746fd6755c0d2956d332252cc6ac04dcd6d4ea442ab51fd39da2 not found: ID does not exist"
Sep 16 12:31:59.879874 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.879840    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"]
Sep 16 12:31:59.887413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.887385    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/distinct-2-w0-739ee7-kserve-cdd98c568-mflk8"]
Sep 16 12:31:59.917597 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.917537    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"]
Sep 16 12:31:59.939099 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:31:59.939065    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/distinct-w0-739ee7-kserve-767d687d55-nd957"]
Sep 16 12:32:00.602046 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:00.602014    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:01.706658 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:01.706622    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"]
Sep 16 12:32:01.707025 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:01.706828    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx" podUID="e3e32a8c-2a69-4693-a45f-709fc95ce47f" containerName="maas-api" containerID="cri-o://42ba498b038be7cea9104c91d8c278c8911b3918445aecd0232df510a10047cd" gracePeriod=30
Sep 16 12:32:01.760502 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:01.760468    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="609f3195-96fa-45aa-842a-80222b4bf8ee" path="/var/lib/kubelet/pods/609f3195-96fa-45aa-842a-80222b4bf8ee/volumes"
Sep 16 12:32:01.761164 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:01.761139    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="626a74de-6993-4a37-8f6e-207ec5d6190c" path="/var/lib/kubelet/pods/626a74de-6993-4a37-8f6e-207ec5d6190c/volumes"
Sep 16 12:32:01.803698 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:01.803664    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:01.860923 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:01.860885    2558 generic.go:356] "Generic (PLEG): container finished" podID="e3e32a8c-2a69-4693-a45f-709fc95ce47f" containerID="42ba498b038be7cea9104c91d8c278c8911b3918445aecd0232df510a10047cd" exitCode=0
Sep 16 12:32:01.860923 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:01.860919    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx" event={"ID":"e3e32a8c-2a69-4693-a45f-709fc95ce47f","Type":"ContainerDied","Data":"42ba498b038be7cea9104c91d8c278c8911b3918445aecd0232df510a10047cd"}
Sep 16 12:32:01.976753 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:01.976725    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:32:02.003196 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.003158    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-metrics-certs\") pod \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") "
Sep 16 12:32:02.003408 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.003255    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-maas-api-tls\") pod \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") "
Sep 16 12:32:02.003408 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.003300    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hsxp\" (UniqueName: \"kubernetes.io/projected/e3e32a8c-2a69-4693-a45f-709fc95ce47f-kube-api-access-2hsxp\") pod \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") "
Sep 16 12:32:02.003408 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.003395    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e32a8c-2a69-4693-a45f-709fc95ce47f-openshift-service-ca\") pod \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\" (UID: \"e3e32a8c-2a69-4693-a45f-709fc95ce47f\") "
Sep 16 12:32:02.004066 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.004031    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e32a8c-2a69-4693-a45f-709fc95ce47f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3e32a8c-2a69-4693-a45f-709fc95ce47f" (UID: "e3e32a8c-2a69-4693-a45f-709fc95ce47f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:02.005509 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.005481    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3e32a8c-2a69-4693-a45f-709fc95ce47f" (UID: "e3e32a8c-2a69-4693-a45f-709fc95ce47f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:02.005663 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.005508    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3e32a8c-2a69-4693-a45f-709fc95ce47f" (UID: "e3e32a8c-2a69-4693-a45f-709fc95ce47f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:02.005663 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.005574    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e32a8c-2a69-4693-a45f-709fc95ce47f-kube-api-access-2hsxp" (OuterVolumeSpecName: "kube-api-access-2hsxp") pod "e3e32a8c-2a69-4693-a45f-709fc95ce47f" (UID: "e3e32a8c-2a69-4693-a45f-709fc95ce47f"). InnerVolumeSpecName "kube-api-access-2hsxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:02.037773 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.037742    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:02.105004 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.104960    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e32a8c-2a69-4693-a45f-709fc95ce47f-openshift-service-ca\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:32:02.105004 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.104997    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-metrics-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:32:02.105004 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.105006    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e32a8c-2a69-4693-a45f-709fc95ce47f-maas-api-tls\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:32:02.105004 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.105014    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hsxp\" (UniqueName: \"kubernetes.io/projected/e3e32a8c-2a69-4693-a45f-709fc95ce47f-kube-api-access-2hsxp\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:32:02.307449 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.307356    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:02.558856 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.558763    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:02.868367 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.868333    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"
Sep 16 12:32:02.868812 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.868365    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx" event={"ID":"e3e32a8c-2a69-4693-a45f-709fc95ce47f","Type":"ContainerDied","Data":"51bfd4e6e89b69d644cfd190ad2ebf69b05d4f5903a2ff3fd6392d45a81a24e3"}
Sep 16 12:32:02.868812 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.868420    2558 scope.go:117] "RemoveContainer" containerID="42ba498b038be7cea9104c91d8c278c8911b3918445aecd0232df510a10047cd"
Sep 16 12:32:02.917207 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.917171    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"]
Sep 16 12:32:02.932467 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.932434    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-739ee7-58d779b4c5-5vkkx"]
Sep 16 12:32:02.964822 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:02.964791    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:03.158924 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:03.158840    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:03.759968 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:03.759932    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e32a8c-2a69-4693-a45f-709fc95ce47f" path="/var/lib/kubelet/pods/e3e32a8c-2a69-4693-a45f-709fc95ce47f/volumes"
Sep 16 12:32:04.285820 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:04.285791    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:05.596776 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:05.596733    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:07.387174 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:07.387144    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:08.885553 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:08.885508    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:10.128281 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:10.128247    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:10.296365 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:10.296333    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:11.678638 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:11.678602    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:13.639729 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:13.639689    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:14.595770 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:14.595734    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:14.779990 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:14.779932    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:14.987237 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:14.987140    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:15.494284 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:15.494250    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:16.738602 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:16.738535    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:18.276347 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:18.276298    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:19.418354 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:19.418320    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:20.593242 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:20.591987    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:21.898037 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:21.897997    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:23.022174 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:23.022140    2558 generic.go:356] "Generic (PLEG): container finished" podID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerID="12bc4f4525bc1aa08aac6b244682db3764fa840736ca5a14c2524fbb47bf0440" exitCode=0
Sep 16 12:32:23.022734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:23.022214    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz" event={"ID":"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da","Type":"ContainerDied","Data":"12bc4f4525bc1aa08aac6b244682db3764fa840736ca5a14c2524fbb47bf0440"}
Sep 16 12:32:23.022734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:23.022725    2558 scope.go:117] "RemoveContainer" containerID="12bc4f4525bc1aa08aac6b244682db3764fa840736ca5a14c2524fbb47bf0440"
Sep 16 12:32:23.024350 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:23.024306    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:32:23.024460 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:23.024408    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:32:23.024510 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:23.024458    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:32:23.128136 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:23.128096    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:24.031085 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:24.031037    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz" event={"ID":"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da","Type":"ContainerStarted","Data":"50cbffae5c44e24f42f3317011d8bc25b7efadb12fdcde0b4638a86177c2341e"}
Sep 16 12:32:24.249190 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:24.249102    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:24.881919 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:24.881880    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:32:24.882133 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:24.882110    2558 patch_prober.go:28] interesting pod/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 16 12:32:24.882196 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:24.882176    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz" podUID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 16 12:32:24.939082 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:24.939046    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:25.072570 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:25.068536    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:25.169388 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:25.169287    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:25.276614 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:25.276579    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:25.881760 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:25.881719    2558 patch_prober.go:28] interesting pod/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 16 12:32:25.881980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:25.881813    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz" podUID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 16 12:32:26.881924 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:26.881882    2558 patch_prober.go:28] interesting pod/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 16 12:32:26.882344 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:26.881976    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz" podUID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 16 12:32:27.882208 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:27.882164    2558 patch_prober.go:28] interesting pod/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 16 12:32:27.882630 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:27.882229    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz" podUID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 16 12:32:28.882913 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:28.882869    2558 patch_prober.go:28] interesting pod/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 16 12:32:28.883364 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:28.882932    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz" podUID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 16 12:32:29.758025 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:29.757993    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:32:29.886772 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:29.886740    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:32:30.084117 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:30.084031    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:32:31.561563 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:31.561481    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:31.562460 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:31.562406    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"]
Sep 16 12:32:31.563064 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:31.562740    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv" podUID="b482b9a7-951c-4a0c-bc34-90aa0c70c632" containerName="payload-processing" containerID="cri-o://9bff68982d0bcf782f9b2296c2a67ebb4edae89ecb24f26877f58ea1a755e9d0" gracePeriod=30
Sep 16 12:32:31.652154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:31.652125    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:31.753372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:31.753336    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:31.833062 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:31.833015    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"
Sep 16 12:32:31.855741 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:31.855704    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:31.939706 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:31.939674    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b482b9a7-951c-4a0c-bc34-90aa0c70c632-plugins-config-volume\") pod \"b482b9a7-951c-4a0c-bc34-90aa0c70c632\" (UID: \"b482b9a7-951c-4a0c-bc34-90aa0c70c632\") "
Sep 16 12:32:31.939897 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:31.939819    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfkcf\" (UniqueName: \"kubernetes.io/projected/b482b9a7-951c-4a0c-bc34-90aa0c70c632-kube-api-access-tfkcf\") pod \"b482b9a7-951c-4a0c-bc34-90aa0c70c632\" (UID: \"b482b9a7-951c-4a0c-bc34-90aa0c70c632\") "
Sep 16 12:32:31.940038 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:31.940015    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b482b9a7-951c-4a0c-bc34-90aa0c70c632-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b482b9a7-951c-4a0c-bc34-90aa0c70c632" (UID: "b482b9a7-951c-4a0c-bc34-90aa0c70c632"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:31.941993 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:31.941963    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b482b9a7-951c-4a0c-bc34-90aa0c70c632-kube-api-access-tfkcf" (OuterVolumeSpecName: "kube-api-access-tfkcf") pod "b482b9a7-951c-4a0c-bc34-90aa0c70c632" (UID: "b482b9a7-951c-4a0c-bc34-90aa0c70c632"). InnerVolumeSpecName "kube-api-access-tfkcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:32.041072 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:32.041030    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfkcf\" (UniqueName: \"kubernetes.io/projected/b482b9a7-951c-4a0c-bc34-90aa0c70c632-kube-api-access-tfkcf\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:32:32.041072 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:32.041064    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b482b9a7-951c-4a0c-bc34-90aa0c70c632-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:32:32.095281 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:32.095244    2558 generic.go:356] "Generic (PLEG): container finished" podID="b482b9a7-951c-4a0c-bc34-90aa0c70c632" containerID="9bff68982d0bcf782f9b2296c2a67ebb4edae89ecb24f26877f58ea1a755e9d0" exitCode=0
Sep 16 12:32:32.095476 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:32.095298    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"
Sep 16 12:32:32.095476 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:32.095332    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv" event={"ID":"b482b9a7-951c-4a0c-bc34-90aa0c70c632","Type":"ContainerDied","Data":"9bff68982d0bcf782f9b2296c2a67ebb4edae89ecb24f26877f58ea1a755e9d0"}
Sep 16 12:32:32.095476 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:32.095428    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv" event={"ID":"b482b9a7-951c-4a0c-bc34-90aa0c70c632","Type":"ContainerDied","Data":"be1f5330ba79bbfc9e876d80b1a9276642fbdd45cca1a6e540b778a47448cbf9"}
Sep 16 12:32:32.095476 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:32.095458    2558 scope.go:117] "RemoveContainer" containerID="9bff68982d0bcf782f9b2296c2a67ebb4edae89ecb24f26877f58ea1a755e9d0"
Sep 16 12:32:32.105632 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:32.105611    2558 scope.go:117] "RemoveContainer" containerID="9bff68982d0bcf782f9b2296c2a67ebb4edae89ecb24f26877f58ea1a755e9d0"
Sep 16 12:32:32.105903 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:32:32.105883    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bff68982d0bcf782f9b2296c2a67ebb4edae89ecb24f26877f58ea1a755e9d0\": container with ID starting with 9bff68982d0bcf782f9b2296c2a67ebb4edae89ecb24f26877f58ea1a755e9d0 not found: ID does not exist" containerID="9bff68982d0bcf782f9b2296c2a67ebb4edae89ecb24f26877f58ea1a755e9d0"
Sep 16 12:32:32.105949 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:32.105912    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bff68982d0bcf782f9b2296c2a67ebb4edae89ecb24f26877f58ea1a755e9d0"} err="failed to get container status \"9bff68982d0bcf782f9b2296c2a67ebb4edae89ecb24f26877f58ea1a755e9d0\": rpc error: code = NotFound desc = could not find container \"9bff68982d0bcf782f9b2296c2a67ebb4edae89ecb24f26877f58ea1a755e9d0\": container with ID starting with 9bff68982d0bcf782f9b2296c2a67ebb4edae89ecb24f26877f58ea1a755e9d0 not found: ID does not exist"
Sep 16 12:32:32.124719 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:32.124681    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"]
Sep 16 12:32:32.129312 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:32.129288    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8641cd-5cc7c446c-b9cxv"]
Sep 16 12:32:32.152420 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:32.152382    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:32.253577 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:32.253526    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:33.107662 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:33.107629    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:33.760808 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:33.760776    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b482b9a7-951c-4a0c-bc34-90aa0c70c632" path="/var/lib/kubelet/pods/b482b9a7-951c-4a0c-bc34-90aa0c70c632/volumes"
Sep 16 12:32:33.948857 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:33.948804    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:34.874083 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:34.874043    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:35.786105 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:35.786052    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:36.710679 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:36.710636    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:37.778330 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:37.778295    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:39.883467 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:39.883428    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:39.987718 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:39.987685    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:40.240644 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:40.240565    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:40.736819 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:40.736780    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:41.911823 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:41.911791    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:43.054285 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:43.054252    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:43.265322 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:43.265268    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:44.465922 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:44.465893    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:45.884304 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:45.884273    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:47.256373 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:47.256338    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:49.029512 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:49.029476    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:55.967669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:55.967621    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:56.213608 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:56.213578    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:56.420946 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:56.420910    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:56.637018 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:56.636975    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:56.922557 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:56.922521    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:57.138730 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:57.138695    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:58.533480 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:32:58.533425    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:00.157227 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:00.157189    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:01.836949 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:01.836915    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:03.395655 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:03.395619    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:04.808195 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:04.808147    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:06.431075 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:06.431041    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:06.652953 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:06.652915    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:06.706484 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:06.706393    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"]
Sep 16 12:33:06.706661 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:06.706641    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz" podUID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerName="istio-proxy" containerID="cri-o://50cbffae5c44e24f42f3317011d8bc25b7efadb12fdcde0b4638a86177c2341e" gracePeriod=30
Sep 16 12:33:06.813051 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:06.813019    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:07.488998 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:07.488959    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:08.359684 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:08.359650    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:08.907965 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:08.907930    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:10.074797 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:10.074763    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:11.291869 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:11.291835    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:11.970093 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:11.970016    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:33:12.055731 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.055695    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-podinfo\") pod \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") "
Sep 16 12:33:12.055731 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.055735    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-token\") pod \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") "
Sep 16 12:33:12.055980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.055764    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-envoy\") pod \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") "
Sep 16 12:33:12.055980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.055813    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-data\") pod \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") "
Sep 16 12:33:12.055980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.055839    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phd2r\" (UniqueName: \"kubernetes.io/projected/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-kube-api-access-phd2r\") pod \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") "
Sep 16 12:33:12.055980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.055854    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-workload-socket\") pod \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") "
Sep 16 12:33:12.055980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.055888    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-credential-socket\") pod \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") "
Sep 16 12:33:12.055980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.055914    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istiod-ca-cert\") pod \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") "
Sep 16 12:33:12.056292 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.056120    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" (UID: "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:12.056292 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.056184    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" (UID: "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:12.056292 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.056198    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-workload-certs\") pod \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\" (UID: \"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da\") "
Sep 16 12:33:12.056463 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.056343    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" (UID: "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:12.056463 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.056385    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" (UID: "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:12.056605 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.056573    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" (UID: "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:12.056680 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.056663    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-data\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:12.056748 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.056683    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-workload-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:12.056748 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.056692    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-credential-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:12.056748 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.056701    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istiod-ca-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:12.056748 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.056718    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-workload-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:12.058146 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.058115    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" (UID: "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:12.058279 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.058148    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-kube-api-access-phd2r" (OuterVolumeSpecName: "kube-api-access-phd2r") pod "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" (UID: "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da"). InnerVolumeSpecName "kube-api-access-phd2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:12.058391 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.058375    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" (UID: "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:33:12.058451 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.058392    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" (UID: "1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:12.158081 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.158043    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-podinfo\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:12.158081 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.158073    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-token\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:12.158081 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.158082    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-istio-envoy\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:12.158081 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.158090    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phd2r\" (UniqueName: \"kubernetes.io/projected/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da-kube-api-access-phd2r\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:12.335808 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.335707    2558 generic.go:356] "Generic (PLEG): container finished" podID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerID="50cbffae5c44e24f42f3317011d8bc25b7efadb12fdcde0b4638a86177c2341e" exitCode=0
Sep 16 12:33:12.335808 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.335772    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"
Sep 16 12:33:12.335808 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.335788    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz" event={"ID":"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da","Type":"ContainerDied","Data":"50cbffae5c44e24f42f3317011d8bc25b7efadb12fdcde0b4638a86177c2341e"}
Sep 16 12:33:12.336388 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.335867    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz" event={"ID":"1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da","Type":"ContainerDied","Data":"d9ceec1617fc86ea6e23f880a3186dcca2da8a5e6688cc6e8d4aed308c5cda8a"}
Sep 16 12:33:12.336388 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.335891    2558 scope.go:117] "RemoveContainer" containerID="50cbffae5c44e24f42f3317011d8bc25b7efadb12fdcde0b4638a86177c2341e"
Sep 16 12:33:12.345909 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.345889    2558 scope.go:117] "RemoveContainer" containerID="12bc4f4525bc1aa08aac6b244682db3764fa840736ca5a14c2524fbb47bf0440"
Sep 16 12:33:12.354204 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.354182    2558 scope.go:117] "RemoveContainer" containerID="50cbffae5c44e24f42f3317011d8bc25b7efadb12fdcde0b4638a86177c2341e"
Sep 16 12:33:12.354556 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:33:12.354510    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50cbffae5c44e24f42f3317011d8bc25b7efadb12fdcde0b4638a86177c2341e\": container with ID starting with 50cbffae5c44e24f42f3317011d8bc25b7efadb12fdcde0b4638a86177c2341e not found: ID does not exist" containerID="50cbffae5c44e24f42f3317011d8bc25b7efadb12fdcde0b4638a86177c2341e"
Sep 16 12:33:12.354615 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.354575    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50cbffae5c44e24f42f3317011d8bc25b7efadb12fdcde0b4638a86177c2341e"} err="failed to get container status \"50cbffae5c44e24f42f3317011d8bc25b7efadb12fdcde0b4638a86177c2341e\": rpc error: code = NotFound desc = could not find container \"50cbffae5c44e24f42f3317011d8bc25b7efadb12fdcde0b4638a86177c2341e\": container with ID starting with 50cbffae5c44e24f42f3317011d8bc25b7efadb12fdcde0b4638a86177c2341e not found: ID does not exist"
Sep 16 12:33:12.354660 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.354617    2558 scope.go:117] "RemoveContainer" containerID="12bc4f4525bc1aa08aac6b244682db3764fa840736ca5a14c2524fbb47bf0440"
Sep 16 12:33:12.354896 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:33:12.354879    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12bc4f4525bc1aa08aac6b244682db3764fa840736ca5a14c2524fbb47bf0440\": container with ID starting with 12bc4f4525bc1aa08aac6b244682db3764fa840736ca5a14c2524fbb47bf0440 not found: ID does not exist" containerID="12bc4f4525bc1aa08aac6b244682db3764fa840736ca5a14c2524fbb47bf0440"
Sep 16 12:33:12.354943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.354902    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12bc4f4525bc1aa08aac6b244682db3764fa840736ca5a14c2524fbb47bf0440"} err="failed to get container status \"12bc4f4525bc1aa08aac6b244682db3764fa840736ca5a14c2524fbb47bf0440\": rpc error: code = NotFound desc = could not find container \"12bc4f4525bc1aa08aac6b244682db3764fa840736ca5a14c2524fbb47bf0440\": container with ID starting with 12bc4f4525bc1aa08aac6b244682db3764fa840736ca5a14c2524fbb47bf0440 not found: ID does not exist"
Sep 16 12:33:12.392047 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.392009    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"]
Sep 16 12:33:12.405157 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.405119    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-352217-openshift-default-bd49f79fd-rc5dz"]
Sep 16 12:33:12.856300 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:12.856266    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:13.156978 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:13.156884    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"]
Sep 16 12:33:13.349627 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:13.349597    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:13.758874 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:13.758838    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" path="/var/lib/kubelet/pods/1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da/volumes"
Sep 16 12:33:14.218252 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:14.218216    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:15.526325 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:15.526292    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:18.113669 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.113594    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:18.166901 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.166862    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"]
Sep 16 12:33:18.167148 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.167121    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj" podUID="5f99096b-29aa-4004-9879-3741bbe7797b" containerName="payload-processing" containerID="cri-o://644835d793fa9c5a2cedc1c3dcf8c3fd0f3f3b51bd6fd2c80f0990769986cae9" gracePeriod=30
Sep 16 12:33:18.179535 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.179502    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"]
Sep 16 12:33:18.179837 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.179786    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2" podUID="560665e9-e61c-4b7a-86cd-54be23492507" containerName="payload-pre-processing" containerID="cri-o://d557f041253618f463fd146ce4aafba2a95771d5894d685f2244a534041959db" gracePeriod=30
Sep 16 12:33:18.249662 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.249478    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:18.373985 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.373903    2558 generic.go:356] "Generic (PLEG): container finished" podID="5f99096b-29aa-4004-9879-3741bbe7797b" containerID="644835d793fa9c5a2cedc1c3dcf8c3fd0f3f3b51bd6fd2c80f0990769986cae9" exitCode=0
Sep 16 12:33:18.373985 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.373971    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj" event={"ID":"5f99096b-29aa-4004-9879-3741bbe7797b","Type":"ContainerDied","Data":"644835d793fa9c5a2cedc1c3dcf8c3fd0f3f3b51bd6fd2c80f0990769986cae9"}
Sep 16 12:33:18.375897 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.375866    2558 generic.go:356] "Generic (PLEG): container finished" podID="560665e9-e61c-4b7a-86cd-54be23492507" containerID="d557f041253618f463fd146ce4aafba2a95771d5894d685f2244a534041959db" exitCode=0
Sep 16 12:33:18.376045 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.375908    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2" event={"ID":"560665e9-e61c-4b7a-86cd-54be23492507","Type":"ContainerDied","Data":"d557f041253618f463fd146ce4aafba2a95771d5894d685f2244a534041959db"}
Sep 16 12:33:18.460121 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.460024    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"
Sep 16 12:33:18.467668 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.467628    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:18.490576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.490530    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"
Sep 16 12:33:18.527075 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.527045    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f99096b-29aa-4004-9879-3741bbe7797b-plugins-config-volume\") pod \"5f99096b-29aa-4004-9879-3741bbe7797b\" (UID: \"5f99096b-29aa-4004-9879-3741bbe7797b\") "
Sep 16 12:33:18.527259 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.527139    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxxn8\" (UniqueName: \"kubernetes.io/projected/5f99096b-29aa-4004-9879-3741bbe7797b-kube-api-access-gxxn8\") pod \"5f99096b-29aa-4004-9879-3741bbe7797b\" (UID: \"5f99096b-29aa-4004-9879-3741bbe7797b\") "
Sep 16 12:33:18.527422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.527396    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f99096b-29aa-4004-9879-3741bbe7797b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f99096b-29aa-4004-9879-3741bbe7797b" (UID: "5f99096b-29aa-4004-9879-3741bbe7797b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:18.529354 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.529327    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f99096b-29aa-4004-9879-3741bbe7797b-kube-api-access-gxxn8" (OuterVolumeSpecName: "kube-api-access-gxxn8") pod "5f99096b-29aa-4004-9879-3741bbe7797b" (UID: "5f99096b-29aa-4004-9879-3741bbe7797b"). InnerVolumeSpecName "kube-api-access-gxxn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:18.628302 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.628217    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zt4q5\" (UniqueName: \"kubernetes.io/projected/560665e9-e61c-4b7a-86cd-54be23492507-kube-api-access-zt4q5\") pod \"560665e9-e61c-4b7a-86cd-54be23492507\" (UID: \"560665e9-e61c-4b7a-86cd-54be23492507\") "
Sep 16 12:33:18.628302 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.628280    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/560665e9-e61c-4b7a-86cd-54be23492507-plugins-config-volume\") pod \"560665e9-e61c-4b7a-86cd-54be23492507\" (UID: \"560665e9-e61c-4b7a-86cd-54be23492507\") "
Sep 16 12:33:18.628515 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.628504    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxxn8\" (UniqueName: \"kubernetes.io/projected/5f99096b-29aa-4004-9879-3741bbe7797b-kube-api-access-gxxn8\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:18.628571 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.628517    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f99096b-29aa-4004-9879-3741bbe7797b-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:18.628717 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.628682    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560665e9-e61c-4b7a-86cd-54be23492507-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "560665e9-e61c-4b7a-86cd-54be23492507" (UID: "560665e9-e61c-4b7a-86cd-54be23492507"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:18.630389 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.630367    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560665e9-e61c-4b7a-86cd-54be23492507-kube-api-access-zt4q5" (OuterVolumeSpecName: "kube-api-access-zt4q5") pod "560665e9-e61c-4b7a-86cd-54be23492507" (UID: "560665e9-e61c-4b7a-86cd-54be23492507"). InnerVolumeSpecName "kube-api-access-zt4q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:18.675970 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.675905    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:18.729087 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.729049    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/560665e9-e61c-4b7a-86cd-54be23492507-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:18.729087 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:18.729079    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt4q5\" (UniqueName: \"kubernetes.io/projected/560665e9-e61c-4b7a-86cd-54be23492507-kube-api-access-zt4q5\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:19.382943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.382907    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"
Sep 16 12:33:19.382943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.382921    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj" event={"ID":"5f99096b-29aa-4004-9879-3741bbe7797b","Type":"ContainerDied","Data":"5aa633b5cbf04fff7eb56d059e32f7bb909f22bede2663f7080451f497a4dffd"}
Sep 16 12:33:19.383494 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.382990    2558 scope.go:117] "RemoveContainer" containerID="644835d793fa9c5a2cedc1c3dcf8c3fd0f3f3b51bd6fd2c80f0990769986cae9"
Sep 16 12:33:19.384412 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.384391    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"
Sep 16 12:33:19.384487 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.384445    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2" event={"ID":"560665e9-e61c-4b7a-86cd-54be23492507","Type":"ContainerDied","Data":"70d91bf6b1ce02511025bbb001ac6e6370806414839f8d9a9a74d9a556e58e15"}
Sep 16 12:33:19.392175 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.392156    2558 scope.go:117] "RemoveContainer" containerID="d557f041253618f463fd146ce4aafba2a95771d5894d685f2244a534041959db"
Sep 16 12:33:19.412847 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.412813    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:19.466305 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.466202    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"]
Sep 16 12:33:19.490271 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.490234    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cadd00-777886d4c7-m2qqj"]
Sep 16 12:33:19.555476 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.555437    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"]
Sep 16 12:33:19.557317 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.557289    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cadd00-7cd6d785f8-x8cw2"]
Sep 16 12:33:19.743268 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.743180    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:19.758334 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.758303    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="560665e9-e61c-4b7a-86cd-54be23492507" path="/var/lib/kubelet/pods/560665e9-e61c-4b7a-86cd-54be23492507/volumes"
Sep 16 12:33:19.758742 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:19.758728    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f99096b-29aa-4004-9879-3741bbe7797b" path="/var/lib/kubelet/pods/5f99096b-29aa-4004-9879-3741bbe7797b/volumes"
Sep 16 12:33:21.542423 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:21.542391    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:22.964069 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:22.964018    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"]
Sep 16 12:33:22.964536 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:22.964230    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q" podUID="04bd7bbf-7580-4df8-a82a-3c49f018386b" containerName="maas-api" containerID="cri-o://5a8437491376fb309c3a8e19264d01008d9a3cf0ffa996195bc49bb6f6f1a32b" gracePeriod=30
Sep 16 12:33:23.233790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.233761    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:33:23.379741 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.379690    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bd7bbf-7580-4df8-a82a-3c49f018386b-metrics-certs\") pod \"04bd7bbf-7580-4df8-a82a-3c49f018386b\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") "
Sep 16 12:33:23.379956 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.379762    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bd7bbf-7580-4df8-a82a-3c49f018386b-openshift-service-ca\") pod \"04bd7bbf-7580-4df8-a82a-3c49f018386b\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") "
Sep 16 12:33:23.379956 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.379801    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5z8p\" (UniqueName: \"kubernetes.io/projected/04bd7bbf-7580-4df8-a82a-3c49f018386b-kube-api-access-w5z8p\") pod \"04bd7bbf-7580-4df8-a82a-3c49f018386b\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") "
Sep 16 12:33:23.379956 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.379862    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bd7bbf-7580-4df8-a82a-3c49f018386b-maas-api-tls\") pod \"04bd7bbf-7580-4df8-a82a-3c49f018386b\" (UID: \"04bd7bbf-7580-4df8-a82a-3c49f018386b\") "
Sep 16 12:33:23.380202 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.380176    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bd7bbf-7580-4df8-a82a-3c49f018386b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "04bd7bbf-7580-4df8-a82a-3c49f018386b" (UID: "04bd7bbf-7580-4df8-a82a-3c49f018386b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:23.381958 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.381933    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bd7bbf-7580-4df8-a82a-3c49f018386b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "04bd7bbf-7580-4df8-a82a-3c49f018386b" (UID: "04bd7bbf-7580-4df8-a82a-3c49f018386b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:33:23.382027 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.381987    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bd7bbf-7580-4df8-a82a-3c49f018386b-kube-api-access-w5z8p" (OuterVolumeSpecName: "kube-api-access-w5z8p") pod "04bd7bbf-7580-4df8-a82a-3c49f018386b" (UID: "04bd7bbf-7580-4df8-a82a-3c49f018386b"). InnerVolumeSpecName "kube-api-access-w5z8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:23.382027 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.382015    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bd7bbf-7580-4df8-a82a-3c49f018386b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "04bd7bbf-7580-4df8-a82a-3c49f018386b" (UID: "04bd7bbf-7580-4df8-a82a-3c49f018386b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:33:23.413832 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.413792    2558 generic.go:356] "Generic (PLEG): container finished" podID="04bd7bbf-7580-4df8-a82a-3c49f018386b" containerID="5a8437491376fb309c3a8e19264d01008d9a3cf0ffa996195bc49bb6f6f1a32b" exitCode=0
Sep 16 12:33:23.413978 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.413844    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"
Sep 16 12:33:23.413978 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.413876    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q" event={"ID":"04bd7bbf-7580-4df8-a82a-3c49f018386b","Type":"ContainerDied","Data":"5a8437491376fb309c3a8e19264d01008d9a3cf0ffa996195bc49bb6f6f1a32b"}
Sep 16 12:33:23.413978 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.413932    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q" event={"ID":"04bd7bbf-7580-4df8-a82a-3c49f018386b","Type":"ContainerDied","Data":"5e417a094f61d0c36e8a8d73c0cd081650dc9f7756aeb0656f6147555771210f"}
Sep 16 12:33:23.413978 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.413954    2558 scope.go:117] "RemoveContainer" containerID="5a8437491376fb309c3a8e19264d01008d9a3cf0ffa996195bc49bb6f6f1a32b"
Sep 16 12:33:23.423832 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.423814    2558 scope.go:117] "RemoveContainer" containerID="5a8437491376fb309c3a8e19264d01008d9a3cf0ffa996195bc49bb6f6f1a32b"
Sep 16 12:33:23.424141 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:33:23.424121    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a8437491376fb309c3a8e19264d01008d9a3cf0ffa996195bc49bb6f6f1a32b\": container with ID starting with 5a8437491376fb309c3a8e19264d01008d9a3cf0ffa996195bc49bb6f6f1a32b not found: ID does not exist" containerID="5a8437491376fb309c3a8e19264d01008d9a3cf0ffa996195bc49bb6f6f1a32b"
Sep 16 12:33:23.424189 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.424152    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8437491376fb309c3a8e19264d01008d9a3cf0ffa996195bc49bb6f6f1a32b"} err="failed to get container status \"5a8437491376fb309c3a8e19264d01008d9a3cf0ffa996195bc49bb6f6f1a32b\": rpc error: code = NotFound desc = could not find container \"5a8437491376fb309c3a8e19264d01008d9a3cf0ffa996195bc49bb6f6f1a32b\": container with ID starting with 5a8437491376fb309c3a8e19264d01008d9a3cf0ffa996195bc49bb6f6f1a32b not found: ID does not exist"
Sep 16 12:33:23.455832 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.455799    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:23.463152 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.463123    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"]
Sep 16 12:33:23.480962 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.480879    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bd7bbf-7580-4df8-a82a-3c49f018386b-metrics-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:23.480962 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.480910    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bd7bbf-7580-4df8-a82a-3c49f018386b-openshift-service-ca\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:23.480962 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.480920    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5z8p\" (UniqueName: \"kubernetes.io/projected/04bd7bbf-7580-4df8-a82a-3c49f018386b-kube-api-access-w5z8p\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:23.480962 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.480930    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bd7bbf-7580-4df8-a82a-3c49f018386b-maas-api-tls\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:23.488709 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.488682    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-68f7cd-7476d59d85-hw52q"]
Sep 16 12:33:23.753074 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.752966    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:23.758678 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.758649    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04bd7bbf-7580-4df8-a82a-3c49f018386b" path="/var/lib/kubelet/pods/04bd7bbf-7580-4df8-a82a-3c49f018386b/volumes"
Sep 16 12:33:23.905755 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:23.905721    2558 scope.go:117] "RemoveContainer" containerID="3227fc57e9b5d1439a62cc834f03e6b3c0ab24ac718b8b6dad339f2f00abe8b2"
Sep 16 12:33:24.033806 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:24.033714    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:24.281792 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:24.281759    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:24.566449 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:24.566418    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:26.124332 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:26.124287    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:28.056322 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.056276    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:28.175865 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.175816    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz" podUID="1bf41d9b-0610-430f-86fd-52c6c8ff0d35" containerName="main" containerID="cri-o://19e08067be0596847dc96dc8c1f3ac021a714a7631863ff0b9fd87aa4c99b842" gracePeriod=45
Sep 16 12:33:28.433757 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.433725    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:33:28.443015 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.442984    2558 generic.go:356] "Generic (PLEG): container finished" podID="1bf41d9b-0610-430f-86fd-52c6c8ff0d35" containerID="19e08067be0596847dc96dc8c1f3ac021a714a7631863ff0b9fd87aa4c99b842" exitCode=0
Sep 16 12:33:28.443162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.443032    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"
Sep 16 12:33:28.443162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.443069    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz" event={"ID":"1bf41d9b-0610-430f-86fd-52c6c8ff0d35","Type":"ContainerDied","Data":"19e08067be0596847dc96dc8c1f3ac021a714a7631863ff0b9fd87aa4c99b842"}
Sep 16 12:33:28.443162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.443112    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz" event={"ID":"1bf41d9b-0610-430f-86fd-52c6c8ff0d35","Type":"ContainerDied","Data":"4f772344da8c5366fa3b7286f93840ed93f38793a46f3109d1ffb347d55d83ea"}
Sep 16 12:33:28.443162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.443131    2558 scope.go:117] "RemoveContainer" containerID="19e08067be0596847dc96dc8c1f3ac021a714a7631863ff0b9fd87aa4c99b842"
Sep 16 12:33:28.451930 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.451912    2558 scope.go:117] "RemoveContainer" containerID="19e08067be0596847dc96dc8c1f3ac021a714a7631863ff0b9fd87aa4c99b842"
Sep 16 12:33:28.452210 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:33:28.452189    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e08067be0596847dc96dc8c1f3ac021a714a7631863ff0b9fd87aa4c99b842\": container with ID starting with 19e08067be0596847dc96dc8c1f3ac021a714a7631863ff0b9fd87aa4c99b842 not found: ID does not exist" containerID="19e08067be0596847dc96dc8c1f3ac021a714a7631863ff0b9fd87aa4c99b842"
Sep 16 12:33:28.452281 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.452223    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e08067be0596847dc96dc8c1f3ac021a714a7631863ff0b9fd87aa4c99b842"} err="failed to get container status \"19e08067be0596847dc96dc8c1f3ac021a714a7631863ff0b9fd87aa4c99b842\": rpc error: code = NotFound desc = could not find container \"19e08067be0596847dc96dc8c1f3ac021a714a7631863ff0b9fd87aa4c99b842\": container with ID starting with 19e08067be0596847dc96dc8c1f3ac021a714a7631863ff0b9fd87aa4c99b842 not found: ID does not exist"
Sep 16 12:33:28.534932 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.534889    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-dshm\") pod \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") "
Sep 16 12:33:28.534932 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.534935    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-tmp-dir\") pod \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") "
Sep 16 12:33:28.535200 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.535033    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw2pt\" (UniqueName: \"kubernetes.io/projected/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-kube-api-access-hw2pt\") pod \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") "
Sep 16 12:33:28.535200 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.535079    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-model-cache\") pod \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") "
Sep 16 12:33:28.535200 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.535109    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-tls-certs\") pod \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") "
Sep 16 12:33:28.535200 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.535140    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-home\") pod \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\" (UID: \"1bf41d9b-0610-430f-86fd-52c6c8ff0d35\") "
Sep 16 12:33:28.535390 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.535229    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bf41d9b-0610-430f-86fd-52c6c8ff0d35" (UID: "1bf41d9b-0610-430f-86fd-52c6c8ff0d35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.535390 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.535341    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bf41d9b-0610-430f-86fd-52c6c8ff0d35" (UID: "1bf41d9b-0610-430f-86fd-52c6c8ff0d35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.535525 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.535502    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-home" (OuterVolumeSpecName: "home") pod "1bf41d9b-0610-430f-86fd-52c6c8ff0d35" (UID: "1bf41d9b-0610-430f-86fd-52c6c8ff0d35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.535614 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.535595    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-tmp-dir\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.535657 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.535620    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-model-cache\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.537054 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.537024    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-dshm" (OuterVolumeSpecName: "dshm") pod "1bf41d9b-0610-430f-86fd-52c6c8ff0d35" (UID: "1bf41d9b-0610-430f-86fd-52c6c8ff0d35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.537511 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.537479    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bf41d9b-0610-430f-86fd-52c6c8ff0d35" (UID: "1bf41d9b-0610-430f-86fd-52c6c8ff0d35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:33:28.537511 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.537484    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-kube-api-access-hw2pt" (OuterVolumeSpecName: "kube-api-access-hw2pt") pod "1bf41d9b-0610-430f-86fd-52c6c8ff0d35" (UID: "1bf41d9b-0610-430f-86fd-52c6c8ff0d35"). InnerVolumeSpecName "kube-api-access-hw2pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:28.636987 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.636902    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw2pt\" (UniqueName: \"kubernetes.io/projected/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-kube-api-access-hw2pt\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.636987 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.636931    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-tls-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.636987 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.636943    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-home\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.636987 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.636951    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf41d9b-0610-430f-86fd-52c6c8ff0d35-dshm\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.799427 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.799391    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"]
Sep 16 12:33:28.832784 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:28.832750    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/distinct-w4-68f7cd-kserve-79c8c67f8f-v8dpz"]
Sep 16 12:33:29.758601 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:29.758558    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf41d9b-0610-430f-86fd-52c6c8ff0d35" path="/var/lib/kubelet/pods/1bf41d9b-0610-430f-86fd-52c6c8ff0d35/volumes"
Sep 16 12:33:31.471330 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:31.471276    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:33.326866 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:33.326835    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:33.450801 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:33.450758    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:33.667730 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:33.667693    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"]
Sep 16 12:33:33.667926 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:33.667904    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy" containerID="cri-o://8a25f3d723baaefb185da99ee0847784cc4aa577b1fbb92ba7a4ac8525c580f3" gracePeriod=30
Sep 16 12:33:33.754692 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:33.754659    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:35.721112 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:35.721081    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:37.214299 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:37.214260    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:33:37.214698 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:37.214335    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:33:37.609936 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:37.609902    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:37.927501 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:37.927404    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:38.090683 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.090618    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:38.254187 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.254098    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"]
Sep 16 12:33:38.254632 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.254306    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerName="istio-proxy" containerID="cri-o://3cb7aa51dd3739fbcf50270248796e7f174c7a41e2f4eb23f2b3ee29a7387032" gracePeriod=30
Sep 16 12:33:38.348297 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.348252    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:38.917388 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.917359    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:33:38.928786 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.928709    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-credential-socket\") pod \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") "
Sep 16 12:33:38.928786 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.928749    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-envoy\") pod \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") "
Sep 16 12:33:38.928786 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.928775    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-workload-certs\") pod \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") "
Sep 16 12:33:38.929036 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.928821    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbrv9\" (UniqueName: \"kubernetes.io/projected/8df6ae1e-9b5c-482d-aa5e-b6d815938648-kube-api-access-gbrv9\") pod \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") "
Sep 16 12:33:38.929036 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.928865    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-token\") pod \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") "
Sep 16 12:33:38.929036 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.928928    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-data\") pod \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") "
Sep 16 12:33:38.929036 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.929002    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istiod-ca-cert\") pod \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") "
Sep 16 12:33:38.929242 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.929031    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8df6ae1e-9b5c-482d-aa5e-b6d815938648" (UID: "8df6ae1e-9b5c-482d-aa5e-b6d815938648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:38.929242 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.929038    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-podinfo\") pod \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") "
Sep 16 12:33:38.929242 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.929113    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-workload-socket\") pod \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\" (UID: \"8df6ae1e-9b5c-482d-aa5e-b6d815938648\") "
Sep 16 12:33:38.929242 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.929136    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8df6ae1e-9b5c-482d-aa5e-b6d815938648" (UID: "8df6ae1e-9b5c-482d-aa5e-b6d815938648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:38.929457 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.929350    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-data" (OuterVolumeSpecName: "istio-data") pod "8df6ae1e-9b5c-482d-aa5e-b6d815938648" (UID: "8df6ae1e-9b5c-482d-aa5e-b6d815938648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:38.929536 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.929519    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-credential-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:38.929617 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.929565    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-workload-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:38.929617 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.929582    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-data\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:38.929617 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.929519    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8df6ae1e-9b5c-482d-aa5e-b6d815938648" (UID: "8df6ae1e-9b5c-482d-aa5e-b6d815938648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:38.929617 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.929521    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8df6ae1e-9b5c-482d-aa5e-b6d815938648" (UID: "8df6ae1e-9b5c-482d-aa5e-b6d815938648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:38.931135 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.931112    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8df6ae1e-9b5c-482d-aa5e-b6d815938648" (UID: "8df6ae1e-9b5c-482d-aa5e-b6d815938648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:38.931135 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.931123    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df6ae1e-9b5c-482d-aa5e-b6d815938648-kube-api-access-gbrv9" (OuterVolumeSpecName: "kube-api-access-gbrv9") pod "8df6ae1e-9b5c-482d-aa5e-b6d815938648" (UID: "8df6ae1e-9b5c-482d-aa5e-b6d815938648"). InnerVolumeSpecName "kube-api-access-gbrv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:38.931296 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.931179    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8df6ae1e-9b5c-482d-aa5e-b6d815938648" (UID: "8df6ae1e-9b5c-482d-aa5e-b6d815938648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:33:38.931296 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:38.931198    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-token" (OuterVolumeSpecName: "istio-token") pod "8df6ae1e-9b5c-482d-aa5e-b6d815938648" (UID: "8df6ae1e-9b5c-482d-aa5e-b6d815938648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:39.030646 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.030613    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istiod-ca-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:39.030646 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.030641    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-podinfo\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:39.030646 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.030651    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-workload-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:39.030888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.030659    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-envoy\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:39.030888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.030670    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbrv9\" (UniqueName: \"kubernetes.io/projected/8df6ae1e-9b5c-482d-aa5e-b6d815938648-kube-api-access-gbrv9\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:39.030888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.030680    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df6ae1e-9b5c-482d-aa5e-b6d815938648-istio-token\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:39.502370 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.502332    2558 generic.go:356] "Generic (PLEG): container finished" podID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerID="8a25f3d723baaefb185da99ee0847784cc4aa577b1fbb92ba7a4ac8525c580f3" exitCode=0
Sep 16 12:33:39.502826 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.502384    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"
Sep 16 12:33:39.502826 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.502409    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" event={"ID":"8df6ae1e-9b5c-482d-aa5e-b6d815938648","Type":"ContainerDied","Data":"8a25f3d723baaefb185da99ee0847784cc4aa577b1fbb92ba7a4ac8525c580f3"}
Sep 16 12:33:39.502826 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.502461    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw" event={"ID":"8df6ae1e-9b5c-482d-aa5e-b6d815938648","Type":"ContainerDied","Data":"cf9055bffcb62ea05114fe70aa397220e93c7ea09f75f34afccf9d10d910cae9"}
Sep 16 12:33:39.502826 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.502484    2558 scope.go:117] "RemoveContainer" containerID="8a25f3d723baaefb185da99ee0847784cc4aa577b1fbb92ba7a4ac8525c580f3"
Sep 16 12:33:39.511862 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.511841    2558 scope.go:117] "RemoveContainer" containerID="841ba8b5c3fb1e37b553e947b08dc9b1346562098c46a0b3ced6f8350f1750b2"
Sep 16 12:33:39.520040 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.520019    2558 scope.go:117] "RemoveContainer" containerID="8a25f3d723baaefb185da99ee0847784cc4aa577b1fbb92ba7a4ac8525c580f3"
Sep 16 12:33:39.520339 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:33:39.520320    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a25f3d723baaefb185da99ee0847784cc4aa577b1fbb92ba7a4ac8525c580f3\": container with ID starting with 8a25f3d723baaefb185da99ee0847784cc4aa577b1fbb92ba7a4ac8525c580f3 not found: ID does not exist" containerID="8a25f3d723baaefb185da99ee0847784cc4aa577b1fbb92ba7a4ac8525c580f3"
Sep 16 12:33:39.520422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.520352    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a25f3d723baaefb185da99ee0847784cc4aa577b1fbb92ba7a4ac8525c580f3"} err="failed to get container status \"8a25f3d723baaefb185da99ee0847784cc4aa577b1fbb92ba7a4ac8525c580f3\": rpc error: code = NotFound desc = could not find container \"8a25f3d723baaefb185da99ee0847784cc4aa577b1fbb92ba7a4ac8525c580f3\": container with ID starting with 8a25f3d723baaefb185da99ee0847784cc4aa577b1fbb92ba7a4ac8525c580f3 not found: ID does not exist"
Sep 16 12:33:39.520422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.520378    2558 scope.go:117] "RemoveContainer" containerID="841ba8b5c3fb1e37b553e947b08dc9b1346562098c46a0b3ced6f8350f1750b2"
Sep 16 12:33:39.520697 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:33:39.520676    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841ba8b5c3fb1e37b553e947b08dc9b1346562098c46a0b3ced6f8350f1750b2\": container with ID starting with 841ba8b5c3fb1e37b553e947b08dc9b1346562098c46a0b3ced6f8350f1750b2 not found: ID does not exist" containerID="841ba8b5c3fb1e37b553e947b08dc9b1346562098c46a0b3ced6f8350f1750b2"
Sep 16 12:33:39.520762 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.520701    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841ba8b5c3fb1e37b553e947b08dc9b1346562098c46a0b3ced6f8350f1750b2"} err="failed to get container status \"841ba8b5c3fb1e37b553e947b08dc9b1346562098c46a0b3ced6f8350f1750b2\": rpc error: code = NotFound desc = could not find container \"841ba8b5c3fb1e37b553e947b08dc9b1346562098c46a0b3ced6f8350f1750b2\": container with ID starting with 841ba8b5c3fb1e37b553e947b08dc9b1346562098c46a0b3ced6f8350f1750b2 not found: ID does not exist"
Sep 16 12:33:39.537494 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.537462    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"]
Sep 16 12:33:39.563968 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.563939    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cadd00-openshift-default-5d66d6d7f4-kggsw"]
Sep 16 12:33:39.680688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.680656    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:39.758041 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:39.757954    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" path="/var/lib/kubelet/pods/8df6ae1e-9b5c-482d-aa5e-b6d815938648/volumes"
Sep 16 12:33:41.133013 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:41.132987    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:43.509448 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.509423    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:33:43.526494 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.526404    2558 generic.go:356] "Generic (PLEG): container finished" podID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerID="3cb7aa51dd3739fbcf50270248796e7f174c7a41e2f4eb23f2b3ee29a7387032" exitCode=0
Sep 16 12:33:43.526494 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.526452    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"
Sep 16 12:33:43.526494 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.526462    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" event={"ID":"04a44004-4f18-4869-9290-faa4f6b4a9cf","Type":"ContainerDied","Data":"3cb7aa51dd3739fbcf50270248796e7f174c7a41e2f4eb23f2b3ee29a7387032"}
Sep 16 12:33:43.526734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.526506    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c" event={"ID":"04a44004-4f18-4869-9290-faa4f6b4a9cf","Type":"ContainerDied","Data":"616a50cac4c2d5619dd6348cc9f3b378dfad944e5a951b1b2a43a0e385ce0b11"}
Sep 16 12:33:43.526734 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.526527    2558 scope.go:117] "RemoveContainer" containerID="3cb7aa51dd3739fbcf50270248796e7f174c7a41e2f4eb23f2b3ee29a7387032"
Sep 16 12:33:43.535184 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.535162    2558 scope.go:117] "RemoveContainer" containerID="e59e49b506d9b959ad6e7cdb45e9d562cb4a881719b4b195ea75eec32e8adde2"
Sep 16 12:33:43.543387 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.543361    2558 scope.go:117] "RemoveContainer" containerID="3cb7aa51dd3739fbcf50270248796e7f174c7a41e2f4eb23f2b3ee29a7387032"
Sep 16 12:33:43.543699 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:33:43.543678    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb7aa51dd3739fbcf50270248796e7f174c7a41e2f4eb23f2b3ee29a7387032\": container with ID starting with 3cb7aa51dd3739fbcf50270248796e7f174c7a41e2f4eb23f2b3ee29a7387032 not found: ID does not exist" containerID="3cb7aa51dd3739fbcf50270248796e7f174c7a41e2f4eb23f2b3ee29a7387032"
Sep 16 12:33:43.543757 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.543709    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb7aa51dd3739fbcf50270248796e7f174c7a41e2f4eb23f2b3ee29a7387032"} err="failed to get container status \"3cb7aa51dd3739fbcf50270248796e7f174c7a41e2f4eb23f2b3ee29a7387032\": rpc error: code = NotFound desc = could not find container \"3cb7aa51dd3739fbcf50270248796e7f174c7a41e2f4eb23f2b3ee29a7387032\": container with ID starting with 3cb7aa51dd3739fbcf50270248796e7f174c7a41e2f4eb23f2b3ee29a7387032 not found: ID does not exist"
Sep 16 12:33:43.543757 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.543729    2558 scope.go:117] "RemoveContainer" containerID="e59e49b506d9b959ad6e7cdb45e9d562cb4a881719b4b195ea75eec32e8adde2"
Sep 16 12:33:43.543984 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:33:43.543966    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e59e49b506d9b959ad6e7cdb45e9d562cb4a881719b4b195ea75eec32e8adde2\": container with ID starting with e59e49b506d9b959ad6e7cdb45e9d562cb4a881719b4b195ea75eec32e8adde2 not found: ID does not exist" containerID="e59e49b506d9b959ad6e7cdb45e9d562cb4a881719b4b195ea75eec32e8adde2"
Sep 16 12:33:43.544030 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.543991    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59e49b506d9b959ad6e7cdb45e9d562cb4a881719b4b195ea75eec32e8adde2"} err="failed to get container status \"e59e49b506d9b959ad6e7cdb45e9d562cb4a881719b4b195ea75eec32e8adde2\": rpc error: code = NotFound desc = could not find container \"e59e49b506d9b959ad6e7cdb45e9d562cb4a881719b4b195ea75eec32e8adde2\": container with ID starting with e59e49b506d9b959ad6e7cdb45e9d562cb4a881719b4b195ea75eec32e8adde2 not found: ID does not exist"
Sep 16 12:33:43.572618 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.572588    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-workload-certs\") pod \"04a44004-4f18-4869-9290-faa4f6b4a9cf\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") "
Sep 16 12:33:43.572821 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.572634    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-data\") pod \"04a44004-4f18-4869-9290-faa4f6b4a9cf\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") "
Sep 16 12:33:43.572821 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.572664    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-podinfo\") pod \"04a44004-4f18-4869-9290-faa4f6b4a9cf\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") "
Sep 16 12:33:43.572821 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.572689    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmvs2\" (UniqueName: \"kubernetes.io/projected/04a44004-4f18-4869-9290-faa4f6b4a9cf-kube-api-access-lmvs2\") pod \"04a44004-4f18-4869-9290-faa4f6b4a9cf\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") "
Sep 16 12:33:43.572821 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.572724    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a44004-4f18-4869-9290-faa4f6b4a9cf-istiod-ca-cert\") pod \"04a44004-4f18-4869-9290-faa4f6b4a9cf\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") "
Sep 16 12:33:43.572821 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.572759    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-token\") pod \"04a44004-4f18-4869-9290-faa4f6b4a9cf\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") "
Sep 16 12:33:43.573088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.572835    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-workload-socket\") pod \"04a44004-4f18-4869-9290-faa4f6b4a9cf\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") "
Sep 16 12:33:43.573088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.572892    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-envoy\") pod \"04a44004-4f18-4869-9290-faa4f6b4a9cf\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") "
Sep 16 12:33:43.573088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.572958    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04a44004-4f18-4869-9290-faa4f6b4a9cf" (UID: "04a44004-4f18-4869-9290-faa4f6b4a9cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:43.573088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.572972    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-data" (OuterVolumeSpecName: "istio-data") pod "04a44004-4f18-4869-9290-faa4f6b4a9cf" (UID: "04a44004-4f18-4869-9290-faa4f6b4a9cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:43.573088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.572976    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-credential-socket\") pod \"04a44004-4f18-4869-9290-faa4f6b4a9cf\" (UID: \"04a44004-4f18-4869-9290-faa4f6b4a9cf\") "
Sep 16 12:33:43.573342 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.573181    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04a44004-4f18-4869-9290-faa4f6b4a9cf" (UID: "04a44004-4f18-4869-9290-faa4f6b4a9cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:43.573458 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.573434    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-credential-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:43.573680 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.573460    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-workload-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:43.573680 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.573476    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-data\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:43.573680 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.573507    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04a44004-4f18-4869-9290-faa4f6b4a9cf" (UID: "04a44004-4f18-4869-9290-faa4f6b4a9cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:43.573680 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.573672    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a44004-4f18-4869-9290-faa4f6b4a9cf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04a44004-4f18-4869-9290-faa4f6b4a9cf" (UID: "04a44004-4f18-4869-9290-faa4f6b4a9cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:43.575668 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.575632    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a44004-4f18-4869-9290-faa4f6b4a9cf-kube-api-access-lmvs2" (OuterVolumeSpecName: "kube-api-access-lmvs2") pod "04a44004-4f18-4869-9290-faa4f6b4a9cf" (UID: "04a44004-4f18-4869-9290-faa4f6b4a9cf"). InnerVolumeSpecName "kube-api-access-lmvs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:43.575790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.575672    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "04a44004-4f18-4869-9290-faa4f6b4a9cf" (UID: "04a44004-4f18-4869-9290-faa4f6b4a9cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:43.575790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.575733    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04a44004-4f18-4869-9290-faa4f6b4a9cf" (UID: "04a44004-4f18-4869-9290-faa4f6b4a9cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:33:43.575790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.575767    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04a44004-4f18-4869-9290-faa4f6b4a9cf" (UID: "04a44004-4f18-4869-9290-faa4f6b4a9cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:43.674645 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.674603    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-envoy\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:43.674645 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.674636    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-podinfo\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:43.674645 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.674648    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmvs2\" (UniqueName: \"kubernetes.io/projected/04a44004-4f18-4869-9290-faa4f6b4a9cf-kube-api-access-lmvs2\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:43.674901 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.674660    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a44004-4f18-4869-9290-faa4f6b4a9cf-istiod-ca-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:43.674901 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.674672    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a44004-4f18-4869-9290-faa4f6b4a9cf-istio-token\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:43.674901 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.674683    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a44004-4f18-4869-9290-faa4f6b4a9cf-workload-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:43.867883 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.867834    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"]
Sep 16 12:33:43.876892 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:43.876856    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-68f7cd-openshift-default-7f8fd578d8-dbt7c"]
Sep 16 12:33:45.303770 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.303733    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:45.315885 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.315855    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"]
Sep 16 12:33:45.316077 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.316052    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr" podUID="456f7c72-de2a-4092-bbc4-3db383337e2e" containerName="payload-processing" containerID="cri-o://f65542eefbdf3e92b7e169ce6f9ce5952e0486e7c048ad174e2e1cdebd19f1f8" gracePeriod=30
Sep 16 12:33:45.539865 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.539826    2558 generic.go:356] "Generic (PLEG): container finished" podID="456f7c72-de2a-4092-bbc4-3db383337e2e" containerID="f65542eefbdf3e92b7e169ce6f9ce5952e0486e7c048ad174e2e1cdebd19f1f8" exitCode=0
Sep 16 12:33:45.540045 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.539869    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr" event={"ID":"456f7c72-de2a-4092-bbc4-3db383337e2e","Type":"ContainerDied","Data":"f65542eefbdf3e92b7e169ce6f9ce5952e0486e7c048ad174e2e1cdebd19f1f8"}
Sep 16 12:33:45.563125 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.563054    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"
Sep 16 12:33:45.594989 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.594953    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwtrk\" (UniqueName: \"kubernetes.io/projected/456f7c72-de2a-4092-bbc4-3db383337e2e-kube-api-access-vwtrk\") pod \"456f7c72-de2a-4092-bbc4-3db383337e2e\" (UID: \"456f7c72-de2a-4092-bbc4-3db383337e2e\") "
Sep 16 12:33:45.595181 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.595037    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456f7c72-de2a-4092-bbc4-3db383337e2e-plugins-config-volume\") pod \"456f7c72-de2a-4092-bbc4-3db383337e2e\" (UID: \"456f7c72-de2a-4092-bbc4-3db383337e2e\") "
Sep 16 12:33:45.595387 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.595362    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456f7c72-de2a-4092-bbc4-3db383337e2e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "456f7c72-de2a-4092-bbc4-3db383337e2e" (UID: "456f7c72-de2a-4092-bbc4-3db383337e2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:45.597107 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.597086    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456f7c72-de2a-4092-bbc4-3db383337e2e-kube-api-access-vwtrk" (OuterVolumeSpecName: "kube-api-access-vwtrk") pod "456f7c72-de2a-4092-bbc4-3db383337e2e" (UID: "456f7c72-de2a-4092-bbc4-3db383337e2e"). InnerVolumeSpecName "kube-api-access-vwtrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:45.673327 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.673288    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:45.696637 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.696601    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456f7c72-de2a-4092-bbc4-3db383337e2e-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:45.696637 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.696631    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwtrk\" (UniqueName: \"kubernetes.io/projected/456f7c72-de2a-4092-bbc4-3db383337e2e-kube-api-access-vwtrk\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:33:45.758061 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.758018    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" path="/var/lib/kubelet/pods/04a44004-4f18-4869-9290-faa4f6b4a9cf/volumes"
Sep 16 12:33:45.851943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:45.851902    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:46.259517 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:46.259429    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:46.439904 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:46.439873    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:46.546475 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:46.546388    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"
Sep 16 12:33:46.546475 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:46.546425    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr" event={"ID":"456f7c72-de2a-4092-bbc4-3db383337e2e","Type":"ContainerDied","Data":"06d73d55bed80a98ad3f563e845f182d3497794beb05fcf4be9a08007d8037f9"}
Sep 16 12:33:46.546721 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:46.546488    2558 scope.go:117] "RemoveContainer" containerID="f65542eefbdf3e92b7e169ce6f9ce5952e0486e7c048ad174e2e1cdebd19f1f8"
Sep 16 12:33:46.592396 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:46.592359    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"]
Sep 16 12:33:46.602160 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:46.602131    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-51ae50-69cc5d5dfc-dqwvr"]
Sep 16 12:33:47.757926 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:47.757892    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="456f7c72-de2a-4092-bbc4-3db383337e2e" path="/var/lib/kubelet/pods/456f7c72-de2a-4092-bbc4-3db383337e2e/volumes"
Sep 16 12:33:48.203335 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:48.203301    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:49.998872 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:49.998833    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:51.090314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:51.090282    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:54.610345 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:54.610304    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:54.861260 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:54.861156    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:55.010854 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:33:55.010807    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:02.231150 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:02.231115    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:02.331596 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:02.331561    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:02.432858 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:02.432824    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:07.766939 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:07.766899    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:11.997370 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:11.997334    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:12.085965 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:12.085931    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:19.970294 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:19.970258    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:20.977628 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:20.977587    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"]
Sep 16 12:34:21.071577 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:21.071524    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:21.201414 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:21.201377    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:33.098906 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:33.098855    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:35.996477 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:35.996413    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg" podUID="f906602e-dd9d-47d5-b03f-d44c09960bc9" containerName="main" containerID="cri-o://49d2c373192974af70d2367e698c269c0e3b1ad06816df325c54b35b7f4b6585" gracePeriod=45
Sep 16 12:34:36.255001 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.254932    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:34:36.383080 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.383037    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-home\") pod \"f906602e-dd9d-47d5-b03f-d44c09960bc9\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") "
Sep 16 12:34:36.383274 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.383093    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-tmp-dir\") pod \"f906602e-dd9d-47d5-b03f-d44c09960bc9\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") "
Sep 16 12:34:36.383274 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.383137    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-model-cache\") pod \"f906602e-dd9d-47d5-b03f-d44c09960bc9\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") "
Sep 16 12:34:36.383274 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.383159    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-dshm\") pod \"f906602e-dd9d-47d5-b03f-d44c09960bc9\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") "
Sep 16 12:34:36.383274 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.383194    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b8d2\" (UniqueName: \"kubernetes.io/projected/f906602e-dd9d-47d5-b03f-d44c09960bc9-kube-api-access-5b8d2\") pod \"f906602e-dd9d-47d5-b03f-d44c09960bc9\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") "
Sep 16 12:34:36.383274 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.383254    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f906602e-dd9d-47d5-b03f-d44c09960bc9-tls-certs\") pod \"f906602e-dd9d-47d5-b03f-d44c09960bc9\" (UID: \"f906602e-dd9d-47d5-b03f-d44c09960bc9\") "
Sep 16 12:34:36.383531 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.383414    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-home" (OuterVolumeSpecName: "home") pod "f906602e-dd9d-47d5-b03f-d44c09960bc9" (UID: "f906602e-dd9d-47d5-b03f-d44c09960bc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:36.383531 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.383466    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f906602e-dd9d-47d5-b03f-d44c09960bc9" (UID: "f906602e-dd9d-47d5-b03f-d44c09960bc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:36.383531 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.383474    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-model-cache" (OuterVolumeSpecName: "model-cache") pod "f906602e-dd9d-47d5-b03f-d44c09960bc9" (UID: "f906602e-dd9d-47d5-b03f-d44c09960bc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:36.385665 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.385623    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-dshm" (OuterVolumeSpecName: "dshm") pod "f906602e-dd9d-47d5-b03f-d44c09960bc9" (UID: "f906602e-dd9d-47d5-b03f-d44c09960bc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:36.385793 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.385712    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f906602e-dd9d-47d5-b03f-d44c09960bc9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f906602e-dd9d-47d5-b03f-d44c09960bc9" (UID: "f906602e-dd9d-47d5-b03f-d44c09960bc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:34:36.385793 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.385767    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f906602e-dd9d-47d5-b03f-d44c09960bc9-kube-api-access-5b8d2" (OuterVolumeSpecName: "kube-api-access-5b8d2") pod "f906602e-dd9d-47d5-b03f-d44c09960bc9" (UID: "f906602e-dd9d-47d5-b03f-d44c09960bc9"). InnerVolumeSpecName "kube-api-access-5b8d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:34:36.484038 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.483998    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-home\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:34:36.484038 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.484031    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-tmp-dir\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:34:36.484038 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.484041    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-model-cache\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:34:36.484038 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.484052    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f906602e-dd9d-47d5-b03f-d44c09960bc9-dshm\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:34:36.484327 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.484061    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b8d2\" (UniqueName: \"kubernetes.io/projected/f906602e-dd9d-47d5-b03f-d44c09960bc9-kube-api-access-5b8d2\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:34:36.484327 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.484070    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f906602e-dd9d-47d5-b03f-d44c09960bc9-tls-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:34:36.792952 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.792915    2558 generic.go:356] "Generic (PLEG): container finished" podID="f906602e-dd9d-47d5-b03f-d44c09960bc9" containerID="49d2c373192974af70d2367e698c269c0e3b1ad06816df325c54b35b7f4b6585" exitCode=0
Sep 16 12:34:36.793172 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.792959    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"
Sep 16 12:34:36.793172 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.793003    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg" event={"ID":"f906602e-dd9d-47d5-b03f-d44c09960bc9","Type":"ContainerDied","Data":"49d2c373192974af70d2367e698c269c0e3b1ad06816df325c54b35b7f4b6585"}
Sep 16 12:34:36.793172 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.793052    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg" event={"ID":"f906602e-dd9d-47d5-b03f-d44c09960bc9","Type":"ContainerDied","Data":"a25b986e411b526718ca6738376651f07433f8a550fc50aeb28f12eadd1a4557"}
Sep 16 12:34:36.793172 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.793073    2558 scope.go:117] "RemoveContainer" containerID="49d2c373192974af70d2367e698c269c0e3b1ad06816df325c54b35b7f4b6585"
Sep 16 12:34:36.802519 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.802498    2558 scope.go:117] "RemoveContainer" containerID="49d2c373192974af70d2367e698c269c0e3b1ad06816df325c54b35b7f4b6585"
Sep 16 12:34:36.802824 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:34:36.802804    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d2c373192974af70d2367e698c269c0e3b1ad06816df325c54b35b7f4b6585\": container with ID starting with 49d2c373192974af70d2367e698c269c0e3b1ad06816df325c54b35b7f4b6585 not found: ID does not exist" containerID="49d2c373192974af70d2367e698c269c0e3b1ad06816df325c54b35b7f4b6585"
Sep 16 12:34:36.802897 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.802836    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d2c373192974af70d2367e698c269c0e3b1ad06816df325c54b35b7f4b6585"} err="failed to get container status \"49d2c373192974af70d2367e698c269c0e3b1ad06816df325c54b35b7f4b6585\": rpc error: code = NotFound desc = could not find container \"49d2c373192974af70d2367e698c269c0e3b1ad06816df325c54b35b7f4b6585\": container with ID starting with 49d2c373192974af70d2367e698c269c0e3b1ad06816df325c54b35b7f4b6585 not found: ID does not exist"
Sep 16 12:34:36.829272 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.829241    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"]
Sep 16 12:34:36.832967 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:36.832938    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/auth-test-model-0e94e5-kserve-7bb5fdd467-rjqjg"]
Sep 16 12:34:37.758709 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:37.758675    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f906602e-dd9d-47d5-b03f-d44c09960bc9" path="/var/lib/kubelet/pods/f906602e-dd9d-47d5-b03f-d44c09960bc9/volumes"
Sep 16 12:34:40.193021 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:40.192980    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:40.272499 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:40.272461    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:56.071638 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.071560    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"]
Sep 16 12:34:56.072012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.071935    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bd7bbf-7580-4df8-a82a-3c49f018386b" containerName="maas-api"
Sep 16 12:34:56.072012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.071946    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bd7bbf-7580-4df8-a82a-3c49f018386b" containerName="maas-api"
Sep 16 12:34:56.072012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.071959    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerName="istio-proxy"
Sep 16 12:34:56.072012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.071965    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerName="istio-proxy"
Sep 16 12:34:56.072012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.071971    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerName="istio-proxy"
Sep 16 12:34:56.072012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.071977    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerName="istio-proxy"
Sep 16 12:34:56.072012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.071984    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerName="istio-proxy"
Sep 16 12:34:56.072012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.071989    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerName="istio-proxy"
Sep 16 12:34:56.072012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.071994    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy"
Sep 16 12:34:56.072012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072000    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy"
Sep 16 12:34:56.072012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072008    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560665e9-e61c-4b7a-86cd-54be23492507" containerName="payload-pre-processing"
Sep 16 12:34:56.072012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072013    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="560665e9-e61c-4b7a-86cd-54be23492507" containerName="payload-pre-processing"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072022    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f906602e-dd9d-47d5-b03f-d44c09960bc9" containerName="main"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072027    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f906602e-dd9d-47d5-b03f-d44c09960bc9" containerName="main"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072033    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e32a8c-2a69-4693-a45f-709fc95ce47f" containerName="maas-api"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072038    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e32a8c-2a69-4693-a45f-709fc95ce47f" containerName="maas-api"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072043    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626a74de-6993-4a37-8f6e-207ec5d6190c" containerName="main"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072048    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="626a74de-6993-4a37-8f6e-207ec5d6190c" containerName="main"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072056    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f99096b-29aa-4004-9879-3741bbe7797b" containerName="payload-processing"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072061    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f99096b-29aa-4004-9879-3741bbe7797b" containerName="payload-processing"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072067    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="456f7c72-de2a-4092-bbc4-3db383337e2e" containerName="payload-processing"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072072    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="456f7c72-de2a-4092-bbc4-3db383337e2e" containerName="payload-processing"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072079    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerName="istio-proxy"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072083    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerName="istio-proxy"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072092    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b482b9a7-951c-4a0c-bc34-90aa0c70c632" containerName="payload-processing"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072097    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b482b9a7-951c-4a0c-bc34-90aa0c70c632" containerName="payload-processing"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072104    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="609f3195-96fa-45aa-842a-80222b4bf8ee" containerName="main"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072108    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="609f3195-96fa-45aa-842a-80222b4bf8ee" containerName="main"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072115    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf41d9b-0610-430f-86fd-52c6c8ff0d35" containerName="main"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072120    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf41d9b-0610-430f-86fd-52c6c8ff0d35" containerName="main"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072170    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="456f7c72-de2a-4092-bbc4-3db383337e2e" containerName="payload-processing"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072176    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="04bd7bbf-7580-4df8-a82a-3c49f018386b" containerName="maas-api"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072181    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="626a74de-6993-4a37-8f6e-207ec5d6190c" containerName="main"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072186    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf41d9b-0610-430f-86fd-52c6c8ff0d35" containerName="main"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072191    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072198    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e32a8c-2a69-4693-a45f-709fc95ce47f" containerName="maas-api"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072203    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="609f3195-96fa-45aa-842a-80222b4bf8ee" containerName="main"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072209    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b482b9a7-951c-4a0c-bc34-90aa0c70c632" containerName="payload-processing"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072215    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerName="istio-proxy"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072220    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerName="istio-proxy"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072225    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="560665e9-e61c-4b7a-86cd-54be23492507" containerName="payload-pre-processing"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072231    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f906602e-dd9d-47d5-b03f-d44c09960bc9" containerName="main"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072238    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f99096b-29aa-4004-9879-3741bbe7797b" containerName="payload-processing"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072242    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a44004-4f18-4869-9290-faa4f6b4a9cf" containerName="istio-proxy"
Sep 16 12:34:56.072372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.072246    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e7568bd-7ee9-419c-b2eb-d4ce3df8e6da" containerName="istio-proxy"
Sep 16 12:34:56.076514 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.076496    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.088077 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.088055    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"default-dockercfg-vbx8f\""
Sep 16 12:34:56.088862 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.088844    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"kube-root-ca.crt\""
Sep 16 12:34:56.088928 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.088846    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"sub-test-model-0e94e5-kserve-self-signed-certs\""
Sep 16 12:34:56.088928 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.088887    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"openshift-service-ca.crt\""
Sep 16 12:34:56.098012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.097980    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"]
Sep 16 12:34:56.132731 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.132682    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:56.158616 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.158536    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-dshm\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.158616 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.158617    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-home\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.158838 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.158695    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqdb\" (UniqueName: \"kubernetes.io/projected/d9e5f417-37da-417c-b992-584f007f3186-kube-api-access-sbqdb\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.158838 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.158736    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e5f417-37da-417c-b992-584f007f3186-tls-certs\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.158838 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.158762    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-tmp-dir\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.158838 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.158794    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-model-cache\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.259686 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.259640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-dshm\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.259686 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.259682    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-home\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.259923 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.259718    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqdb\" (UniqueName: \"kubernetes.io/projected/d9e5f417-37da-417c-b992-584f007f3186-kube-api-access-sbqdb\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.259923 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.259747    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e5f417-37da-417c-b992-584f007f3186-tls-certs\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.259923 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.259765    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-tmp-dir\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.259923 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.259797    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-model-cache\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.260230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.260204    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-home\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.260373 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.260240    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-model-cache\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.260373 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.260260    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-tmp-dir\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.262046 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.262023    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-dshm\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.262223 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.262203    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e5f417-37da-417c-b992-584f007f3186-tls-certs\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.269466 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.269428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqdb\" (UniqueName: \"kubernetes.io/projected/d9e5f417-37da-417c-b992-584f007f3186-kube-api-access-sbqdb\") pod \"sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.385806 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.385767    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:34:56.659559 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.659520    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"]
Sep 16 12:34:56.661471 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:34:56.661443    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9e5f417_37da_417c_b992_584f007f3186.slice/crio-dde54482f49bd6d019fd64e44001f4e35fe72b50db72e469a731965dcebe5652 WatchSource:0}: Error finding container dde54482f49bd6d019fd64e44001f4e35fe72b50db72e469a731965dcebe5652: Status 404 returned error can't find the container with id dde54482f49bd6d019fd64e44001f4e35fe72b50db72e469a731965dcebe5652
Sep 16 12:34:56.889848 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.889807    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c" event={"ID":"d9e5f417-37da-417c-b992-584f007f3186","Type":"ContainerStarted","Data":"d6ff0c2c32cffd7c12afee8589001cabb6283033d6a63a4db49f2165d204da24"}
Sep 16 12:34:56.890021 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.889859    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c" event={"ID":"d9e5f417-37da-417c-b992-584f007f3186","Type":"ContainerStarted","Data":"dde54482f49bd6d019fd64e44001f4e35fe72b50db72e469a731965dcebe5652"}
Sep 16 12:34:56.911329 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:34:56.911213    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c" podStartSLOduration=0.911196831 podStartE2EDuration="911.196831ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:34:56.909280294 +0000 UTC m=+1893.663239025" watchObservedRunningTime="2026-09-16 12:34:56.911196831 +0000 UTC m=+1893.665155562"
Sep 16 12:35:06.386066 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:06.386026    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:35:06.386469 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:06.386122    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:35:06.398872 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:06.398837    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:35:06.951216 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:06.951183    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:35:37.113619 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:37.113584    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:38.589888 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:38.589857    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:40.373679 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:40.373637    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:40.456584 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:40.456529    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:43.216844 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:43.216798    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:44.148983 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:44.148942    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"]
Sep 16 12:35:44.178579 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:44.178533    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:57.151804 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:57.151772    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:59.168401 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.168328    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c" podUID="d9e5f417-37da-417c-b992-584f007f3186" containerName="main" containerID="cri-o://d6ff0c2c32cffd7c12afee8589001cabb6283033d6a63a4db49f2165d204da24" gracePeriod=45
Sep 16 12:35:59.417101 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.417074    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:35:59.545772 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.545684    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-model-cache\") pod \"d9e5f417-37da-417c-b992-584f007f3186\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") "
Sep 16 12:35:59.545772 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.545755    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e5f417-37da-417c-b992-584f007f3186-tls-certs\") pod \"d9e5f417-37da-417c-b992-584f007f3186\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") "
Sep 16 12:35:59.545982 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.545789    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-home\") pod \"d9e5f417-37da-417c-b992-584f007f3186\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") "
Sep 16 12:35:59.545982 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.545812    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-tmp-dir\") pod \"d9e5f417-37da-417c-b992-584f007f3186\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") "
Sep 16 12:35:59.545982 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.545875    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-dshm\") pod \"d9e5f417-37da-417c-b992-584f007f3186\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") "
Sep 16 12:35:59.545982 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.545923    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbqdb\" (UniqueName: \"kubernetes.io/projected/d9e5f417-37da-417c-b992-584f007f3186-kube-api-access-sbqdb\") pod \"d9e5f417-37da-417c-b992-584f007f3186\" (UID: \"d9e5f417-37da-417c-b992-584f007f3186\") "
Sep 16 12:35:59.545982 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.545943    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9e5f417-37da-417c-b992-584f007f3186" (UID: "d9e5f417-37da-417c-b992-584f007f3186"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:59.546252 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.546067    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-home" (OuterVolumeSpecName: "home") pod "d9e5f417-37da-417c-b992-584f007f3186" (UID: "d9e5f417-37da-417c-b992-584f007f3186"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:59.546252 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.546118    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9e5f417-37da-417c-b992-584f007f3186" (UID: "d9e5f417-37da-417c-b992-584f007f3186"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:59.546252 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.546231    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-home\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:35:59.546252 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.546247    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-tmp-dir\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:35:59.546396 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.546260    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-model-cache\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:35:59.548034 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.548000    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-dshm" (OuterVolumeSpecName: "dshm") pod "d9e5f417-37da-417c-b992-584f007f3186" (UID: "d9e5f417-37da-417c-b992-584f007f3186"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:59.548156 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.548037    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e5f417-37da-417c-b992-584f007f3186-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9e5f417-37da-417c-b992-584f007f3186" (UID: "d9e5f417-37da-417c-b992-584f007f3186"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:35:59.548156 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.548050    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e5f417-37da-417c-b992-584f007f3186-kube-api-access-sbqdb" (OuterVolumeSpecName: "kube-api-access-sbqdb") pod "d9e5f417-37da-417c-b992-584f007f3186" (UID: "d9e5f417-37da-417c-b992-584f007f3186"). InnerVolumeSpecName "kube-api-access-sbqdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:35:59.647006 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.646962    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e5f417-37da-417c-b992-584f007f3186-dshm\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:35:59.647006 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.647000    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbqdb\" (UniqueName: \"kubernetes.io/projected/d9e5f417-37da-417c-b992-584f007f3186-kube-api-access-sbqdb\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:35:59.647006 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:35:59.647016    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e5f417-37da-417c-b992-584f007f3186-tls-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:36:00.196042 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:00.196006    2558 generic.go:356] "Generic (PLEG): container finished" podID="d9e5f417-37da-417c-b992-584f007f3186" containerID="d6ff0c2c32cffd7c12afee8589001cabb6283033d6a63a4db49f2165d204da24" exitCode=0
Sep 16 12:36:00.196541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:00.196057    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"
Sep 16 12:36:00.196541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:00.196088    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c" event={"ID":"d9e5f417-37da-417c-b992-584f007f3186","Type":"ContainerDied","Data":"d6ff0c2c32cffd7c12afee8589001cabb6283033d6a63a4db49f2165d204da24"}
Sep 16 12:36:00.196541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:00.196137    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c" event={"ID":"d9e5f417-37da-417c-b992-584f007f3186","Type":"ContainerDied","Data":"dde54482f49bd6d019fd64e44001f4e35fe72b50db72e469a731965dcebe5652"}
Sep 16 12:36:00.196541 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:00.196154    2558 scope.go:117] "RemoveContainer" containerID="d6ff0c2c32cffd7c12afee8589001cabb6283033d6a63a4db49f2165d204da24"
Sep 16 12:36:00.204697 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:00.204677    2558 scope.go:117] "RemoveContainer" containerID="d6ff0c2c32cffd7c12afee8589001cabb6283033d6a63a4db49f2165d204da24"
Sep 16 12:36:00.204983 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:36:00.204965    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6ff0c2c32cffd7c12afee8589001cabb6283033d6a63a4db49f2165d204da24\": container with ID starting with d6ff0c2c32cffd7c12afee8589001cabb6283033d6a63a4db49f2165d204da24 not found: ID does not exist" containerID="d6ff0c2c32cffd7c12afee8589001cabb6283033d6a63a4db49f2165d204da24"
Sep 16 12:36:00.205039 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:00.204992    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6ff0c2c32cffd7c12afee8589001cabb6283033d6a63a4db49f2165d204da24"} err="failed to get container status \"d6ff0c2c32cffd7c12afee8589001cabb6283033d6a63a4db49f2165d204da24\": rpc error: code = NotFound desc = could not find container \"d6ff0c2c32cffd7c12afee8589001cabb6283033d6a63a4db49f2165d204da24\": container with ID starting with d6ff0c2c32cffd7c12afee8589001cabb6283033d6a63a4db49f2165d204da24 not found: ID does not exist"
Sep 16 12:36:00.217161 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:00.217130    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"]
Sep 16 12:36:00.225380 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:00.225352    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/sub-test-model-0e94e5-kserve-f94bc98d7-ddg2c"]
Sep 16 12:36:01.758024 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:01.757987    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e5f417-37da-417c-b992-584f007f3186" path="/var/lib/kubelet/pods/d9e5f417-37da-417c-b992-584f007f3186/volumes"
Sep 16 12:36:15.953727 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.953693    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"]
Sep 16 12:36:15.954268 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.954246    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e5f417-37da-417c-b992-584f007f3186" containerName="main"
Sep 16 12:36:15.954354 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.954271    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e5f417-37da-417c-b992-584f007f3186" containerName="main"
Sep 16 12:36:15.954354 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.954289    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy"
Sep 16 12:36:15.954354 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.954299    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy"
Sep 16 12:36:15.954684 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.954389    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8df6ae1e-9b5c-482d-aa5e-b6d815938648" containerName="istio-proxy"
Sep 16 12:36:15.954684 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.954402    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e5f417-37da-417c-b992-584f007f3186" containerName="main"
Sep 16 12:36:15.957450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.957426    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:15.961156 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.961132    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a9008\"/\"openshift-service-ca.crt\""
Sep 16 12:36:15.961833 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.961816    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a9008\"/\"kube-root-ca.crt\""
Sep 16 12:36:15.961946 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.961848    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a9008\"/\"default-dockercfg-stkpb\""
Sep 16 12:36:15.962012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.961949    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a9008\"/\"rate-test-model-0a9008-kserve-self-signed-certs\""
Sep 16 12:36:15.972190 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.972118    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"]
Sep 16 12:36:15.982660 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:15.982614    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:16.099969 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.099932    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-tmp-dir\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.100158 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.099976    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-model-cache\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.100158 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.100063    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca93dd0-4a59-4c06-9e22-57617168eb95-tls-certs\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.100158 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.100133    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-home\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.100158 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.100149    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-dshm\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.100312 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.100188    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px25s\" (UniqueName: \"kubernetes.io/projected/bca93dd0-4a59-4c06-9e22-57617168eb95-kube-api-access-px25s\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.200720 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.200667    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-home\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.200720 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.200719    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-dshm\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.200997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.200761    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px25s\" (UniqueName: \"kubernetes.io/projected/bca93dd0-4a59-4c06-9e22-57617168eb95-kube-api-access-px25s\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.200997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.200801    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-tmp-dir\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.200997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.200833    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-model-cache\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.200997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.200907    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca93dd0-4a59-4c06-9e22-57617168eb95-tls-certs\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.201222 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.201195    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-home\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.201285 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.201225    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-tmp-dir\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.201285 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.201270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-model-cache\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.203150 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.203121    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-dshm\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.203387 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.203372    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca93dd0-4a59-4c06-9e22-57617168eb95-tls-certs\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.209744 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.209680    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px25s\" (UniqueName: \"kubernetes.io/projected/bca93dd0-4a59-4c06-9e22-57617168eb95-kube-api-access-px25s\") pod \"rate-test-model-0a9008-kserve-54479c799f-zshh4\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.266396 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.266338    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:16.543208 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:16.543082    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"]
Sep 16 12:36:16.545971 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:36:16.545938    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbca93dd0_4a59_4c06_9e22_57617168eb95.slice/crio-64fe003ff948f8229f806b3c1b84af42eb24aab0d69ee05328bb0b829ef6ab12 WatchSource:0}: Error finding container 64fe003ff948f8229f806b3c1b84af42eb24aab0d69ee05328bb0b829ef6ab12: Status 404 returned error can't find the container with id 64fe003ff948f8229f806b3c1b84af42eb24aab0d69ee05328bb0b829ef6ab12
Sep 16 12:36:17.280690 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:17.280646    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4" event={"ID":"bca93dd0-4a59-4c06-9e22-57617168eb95","Type":"ContainerStarted","Data":"25883ff457e77be11bfbf81e4dc71f557181c2650b7749a30751bdac980eac22"}
Sep 16 12:36:17.281096 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:17.280712    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4" event={"ID":"bca93dd0-4a59-4c06-9e22-57617168eb95","Type":"ContainerStarted","Data":"64fe003ff948f8229f806b3c1b84af42eb24aab0d69ee05328bb0b829ef6ab12"}
Sep 16 12:36:17.300216 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:17.300159    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4" podStartSLOduration=2.300141257 podStartE2EDuration="2.300141257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:36:17.298824829 +0000 UTC m=+1974.052783560" watchObservedRunningTime="2026-09-16 12:36:17.300141257 +0000 UTC m=+1974.054099989"
Sep 16 12:36:26.266512 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:26.266469    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:26.266927 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:26.266533    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:26.279314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:26.279279    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:26.332242 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:26.332211    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:36:36.599315 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:36:36.599275    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:11.243940 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:11.243901    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"]
Sep 16 12:37:11.262384 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:11.262350    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:12.284964 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:12.284929    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:12.778052 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:12.778016    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:13.033223 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.033139    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"]
Sep 16 12:37:13.036739 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.036723    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.039639 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.039618    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-869204-openshift-default-dockercfg-xhjrb\""
Sep 16 12:37:13.050946 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.050920    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"]
Sep 16 12:37:13.065225 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.065182    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:13.131038 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.130998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-workload-certs\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.131219 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.131046    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-token\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.131219 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.131106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-credential-socket\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.131219 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.131141    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01c434a-188c-4cc4-a835-1fbf833440d5-istiod-ca-cert\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.131219 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.131166    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-workload-socket\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.131219 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.131193    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-podinfo\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.131495 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.131300    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhtfp\" (UniqueName: \"kubernetes.io/projected/a01c434a-188c-4cc4-a835-1fbf833440d5-kube-api-access-zhtfp\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.131495 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.131342    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-envoy\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.131495 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.131417    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-data\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.232891 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.232846    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01c434a-188c-4cc4-a835-1fbf833440d5-istiod-ca-cert\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.232891 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.232895    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-workload-socket\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.233170 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.232927    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-podinfo\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.233170 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.232968    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhtfp\" (UniqueName: \"kubernetes.io/projected/a01c434a-188c-4cc4-a835-1fbf833440d5-kube-api-access-zhtfp\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.233170 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.232993    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-envoy\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.233170 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.233050    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-data\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.233170 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.233089    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-workload-certs\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.233170 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.233118    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-token\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.233170 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.233163    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-credential-socket\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.233568 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.233385    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-workload-socket\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.233568 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.233516    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-data\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.233568 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.233525    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-credential-socket\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.233747 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.233683    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-workload-certs\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.233803 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.233742    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01c434a-188c-4cc4-a835-1fbf833440d5-istiod-ca-cert\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.235334 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.235315    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-envoy\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.235684 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.235663    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-podinfo\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.242266 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.242233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-token\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.242391 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.242374    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhtfp\" (UniqueName: \"kubernetes.io/projected/a01c434a-188c-4cc4-a835-1fbf833440d5-kube-api-access-zhtfp\") pod \"e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") " pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.347284 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.347248    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:13.454781 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.454752    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:13.624103 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.624078    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"]
Sep 16 12:37:13.626131 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:37:13.626101    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01c434a_188c_4cc4_a835_1fbf833440d5.slice/crio-562da8b9c42ce49926712580d9e2739e32e1b9fe81848ffa0a97d036823c990b WatchSource:0}: Error finding container 562da8b9c42ce49926712580d9e2739e32e1b9fe81848ffa0a97d036823c990b: Status 404 returned error can't find the container with id 562da8b9c42ce49926712580d9e2739e32e1b9fe81848ffa0a97d036823c990b
Sep 16 12:37:13.628468 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.628424    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:37:13.628582 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.628514    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:37:13.628635 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:13.628577    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:37:14.553432 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:14.553389    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" event={"ID":"a01c434a-188c-4cc4-a835-1fbf833440d5","Type":"ContainerStarted","Data":"d89d2ad520e77bd30653c3bced80c6b66fd3f5f7e2e990f78407d3d0f2156dcd"}
Sep 16 12:37:14.553870 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:14.553451    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" event={"ID":"a01c434a-188c-4cc4-a835-1fbf833440d5","Type":"ContainerStarted","Data":"562da8b9c42ce49926712580d9e2739e32e1b9fe81848ffa0a97d036823c990b"}
Sep 16 12:37:14.577390 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:14.577328    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" podStartSLOduration=1.577308667 podStartE2EDuration="1.577308667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:37:14.575270339 +0000 UTC m=+2031.329229071" watchObservedRunningTime="2026-09-16 12:37:14.577308667 +0000 UTC m=+2031.331267397"
Sep 16 12:37:15.348124 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:15.348080    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:15.353327 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:15.353300    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:15.558618 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:15.558584    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:15.559654 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:15.559633    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:15.854637 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:15.854606    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:16.573456 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:16.573413    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:16.675623 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:16.675581    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:16.974953 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:16.974912    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:18.216533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:18.216498    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:18.432749 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:18.432704    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:18.934604 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:18.934566    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"]
Sep 16 12:37:18.938239 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:18.938220    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:18.940586 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:18.940541    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-869204-serving-cert\""
Sep 16 12:37:18.953696 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:18.953664    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"]
Sep 16 12:37:18.992763 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:18.992722    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s5rp\" (UniqueName: \"kubernetes.io/projected/3a1da320-b39d-428f-a1ea-385d9bd1577a-kube-api-access-5s5rp\") pod \"maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:18.992939 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:18.992787    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-metrics-certs\") pod \"maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:18.992939 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:18.992830    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1da320-b39d-428f-a1ea-385d9bd1577a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:18.993015 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:18.992937    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:19.019464 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.019428    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:19.093861 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.093819    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:19.094167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.093921    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s5rp\" (UniqueName: \"kubernetes.io/projected/3a1da320-b39d-428f-a1ea-385d9bd1577a-kube-api-access-5s5rp\") pod \"maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:19.094167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.093970    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-metrics-certs\") pod \"maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:19.094167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.094027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1da320-b39d-428f-a1ea-385d9bd1577a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:19.094517 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:37:19.094496    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-serving-cert: secret "maas-api-e2e-ipp-a-869204-serving-cert" not found
Sep 16 12:37:19.094693 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:37:19.094673    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-maas-api-tls podName:3a1da320-b39d-428f-a1ea-385d9bd1577a nodeName:}" failed. No retries permitted until 2026-09-16 12:37:19.594650955 +0000 UTC m=+2036.348609678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-maas-api-tls") pod "maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d" (UID: "3a1da320-b39d-428f-a1ea-385d9bd1577a") : secret "maas-api-e2e-ipp-a-869204-serving-cert" not found
Sep 16 12:37:19.095210 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.095058    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1da320-b39d-428f-a1ea-385d9bd1577a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:19.098882 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.098826    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-metrics-certs\") pod \"maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:19.110531 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.110502    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s5rp\" (UniqueName: \"kubernetes.io/projected/3a1da320-b39d-428f-a1ea-385d9bd1577a-kube-api-access-5s5rp\") pod \"maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:19.132294 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.132261    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:19.222250 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.222164    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:19.519836 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.519735    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:19.598670 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.598634    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:19.601158 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.601122    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:19.621830 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.621799    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:19.681507 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.681470    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"]
Sep 16 12:37:19.685156 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.685140    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"
Sep 16 12:37:19.689220 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.689177    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-869204\""
Sep 16 12:37:19.689465 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.689451    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-869204-dockercfg-2vdcq\""
Sep 16 12:37:19.709096 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.709060    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"]
Sep 16 12:37:19.800618 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.800505    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf7908c-8e7c-4c5a-802b-a6bceb64075a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz\" (UID: \"baf7908c-8e7c-4c5a-802b-a6bceb64075a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"
Sep 16 12:37:19.800772 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.800647    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs4xd\" (UniqueName: \"kubernetes.io/projected/baf7908c-8e7c-4c5a-802b-a6bceb64075a-kube-api-access-fs4xd\") pod \"payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz\" (UID: \"baf7908c-8e7c-4c5a-802b-a6bceb64075a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"
Sep 16 12:37:19.848960 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.848929    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:19.902156 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.902093    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4xd\" (UniqueName: \"kubernetes.io/projected/baf7908c-8e7c-4c5a-802b-a6bceb64075a-kube-api-access-fs4xd\") pod \"payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz\" (UID: \"baf7908c-8e7c-4c5a-802b-a6bceb64075a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"
Sep 16 12:37:19.902303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.902272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf7908c-8e7c-4c5a-802b-a6bceb64075a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz\" (UID: \"baf7908c-8e7c-4c5a-802b-a6bceb64075a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"
Sep 16 12:37:19.903293 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.903243    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf7908c-8e7c-4c5a-802b-a6bceb64075a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz\" (UID: \"baf7908c-8e7c-4c5a-802b-a6bceb64075a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"
Sep 16 12:37:19.912455 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.912427    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4xd\" (UniqueName: \"kubernetes.io/projected/baf7908c-8e7c-4c5a-802b-a6bceb64075a-kube-api-access-fs4xd\") pod \"payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz\" (UID: \"baf7908c-8e7c-4c5a-802b-a6bceb64075a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"
Sep 16 12:37:19.995396 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:19.995364    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"
Sep 16 12:37:20.127349 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:20.127325    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"]
Sep 16 12:37:20.129496 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:37:20.129468    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1da320_b39d_428f_a1ea_385d9bd1577a.slice/crio-3df271ed8d738f05060aae48be4bd82fa4159c6e39b3d89710cdaa842322d92f WatchSource:0}: Error finding container 3df271ed8d738f05060aae48be4bd82fa4159c6e39b3d89710cdaa842322d92f: Status 404 returned error can't find the container with id 3df271ed8d738f05060aae48be4bd82fa4159c6e39b3d89710cdaa842322d92f
Sep 16 12:37:20.130978 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:20.130950    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:37:20.272425 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:20.272387    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"]
Sep 16 12:37:20.275353 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:37:20.275324    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf7908c_8e7c_4c5a_802b_a6bceb64075a.slice/crio-a889190b0a55b093842f115bd20abfc5ed199d1b6f19a0ac7206fa1237722357 WatchSource:0}: Error finding container a889190b0a55b093842f115bd20abfc5ed199d1b6f19a0ac7206fa1237722357: Status 404 returned error can't find the container with id a889190b0a55b093842f115bd20abfc5ed199d1b6f19a0ac7206fa1237722357
Sep 16 12:37:20.585296 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:20.585200    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d" event={"ID":"3a1da320-b39d-428f-a1ea-385d9bd1577a","Type":"ContainerStarted","Data":"3df271ed8d738f05060aae48be4bd82fa4159c6e39b3d89710cdaa842322d92f"}
Sep 16 12:37:20.586843 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:20.586815    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz" event={"ID":"baf7908c-8e7c-4c5a-802b-a6bceb64075a","Type":"ContainerStarted","Data":"fb772d6d9a5b7e475d6e918eae54dae68557949a9441e83b67e764a2b9139b6a"}
Sep 16 12:37:20.586960 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:20.586858    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz" event={"ID":"baf7908c-8e7c-4c5a-802b-a6bceb64075a","Type":"ContainerStarted","Data":"a889190b0a55b093842f115bd20abfc5ed199d1b6f19a0ac7206fa1237722357"}
Sep 16 12:37:20.587020 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:20.586961    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"
Sep 16 12:37:20.605528 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:20.605475    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz" podStartSLOduration=1.605453492 podStartE2EDuration="1.605453492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:37:20.603588902 +0000 UTC m=+2037.357547633" watchObservedRunningTime="2026-09-16 12:37:20.605453492 +0000 UTC m=+2037.359412219"
Sep 16 12:37:21.593061 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:21.593023    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d" event={"ID":"3a1da320-b39d-428f-a1ea-385d9bd1577a","Type":"ContainerStarted","Data":"2764ecbf79c3a2e4d67044ad03c1591a43fb12e7b24c7796a777e3f370a97c36"}
Sep 16 12:37:21.593472 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:21.593280    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:21.613803 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:21.613744    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d" podStartSLOduration=3.138815356 podStartE2EDuration="3.613726526s" totalImagesPullingTime="474.91117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:37:21.611961878 +0000 UTC m=+2038.365920611" watchObservedRunningTime="2026-09-16 12:37:21.613726526 +0000 UTC m=+2038.367685291"
Sep 16 12:37:21.960656 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:21.960541    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:26.264279 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.264206    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4" podUID="bca93dd0-4a59-4c06-9e22-57617168eb95" containerName="main" containerID="cri-o://25883ff457e77be11bfbf81e4dc71f557181c2650b7749a30751bdac980eac22" gracePeriod=45
Sep 16 12:37:26.321370 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.321329    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4" podUID="bca93dd0-4a59-4c06-9e22-57617168eb95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 16 12:37:26.543347 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.543321    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:37:26.618749 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.618710    2558 generic.go:356] "Generic (PLEG): container finished" podID="bca93dd0-4a59-4c06-9e22-57617168eb95" containerID="25883ff457e77be11bfbf81e4dc71f557181c2650b7749a30751bdac980eac22" exitCode=0
Sep 16 12:37:26.618749 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.618755    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"
Sep 16 12:37:26.618980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.618800    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4" event={"ID":"bca93dd0-4a59-4c06-9e22-57617168eb95","Type":"ContainerDied","Data":"25883ff457e77be11bfbf81e4dc71f557181c2650b7749a30751bdac980eac22"}
Sep 16 12:37:26.618980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.618842    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4" event={"ID":"bca93dd0-4a59-4c06-9e22-57617168eb95","Type":"ContainerDied","Data":"64fe003ff948f8229f806b3c1b84af42eb24aab0d69ee05328bb0b829ef6ab12"}
Sep 16 12:37:26.618980 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.618859    2558 scope.go:117] "RemoveContainer" containerID="25883ff457e77be11bfbf81e4dc71f557181c2650b7749a30751bdac980eac22"
Sep 16 12:37:26.627267 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.627246    2558 scope.go:117] "RemoveContainer" containerID="25883ff457e77be11bfbf81e4dc71f557181c2650b7749a30751bdac980eac22"
Sep 16 12:37:26.627537 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:37:26.627518    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25883ff457e77be11bfbf81e4dc71f557181c2650b7749a30751bdac980eac22\": container with ID starting with 25883ff457e77be11bfbf81e4dc71f557181c2650b7749a30751bdac980eac22 not found: ID does not exist" containerID="25883ff457e77be11bfbf81e4dc71f557181c2650b7749a30751bdac980eac22"
Sep 16 12:37:26.627637 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.627561    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25883ff457e77be11bfbf81e4dc71f557181c2650b7749a30751bdac980eac22"} err="failed to get container status \"25883ff457e77be11bfbf81e4dc71f557181c2650b7749a30751bdac980eac22\": rpc error: code = NotFound desc = could not find container \"25883ff457e77be11bfbf81e4dc71f557181c2650b7749a30751bdac980eac22\": container with ID starting with 25883ff457e77be11bfbf81e4dc71f557181c2650b7749a30751bdac980eac22 not found: ID does not exist"
Sep 16 12:37:26.675785 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.675730    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca93dd0-4a59-4c06-9e22-57617168eb95-tls-certs\") pod \"bca93dd0-4a59-4c06-9e22-57617168eb95\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") "
Sep 16 12:37:26.675785 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.675787    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px25s\" (UniqueName: \"kubernetes.io/projected/bca93dd0-4a59-4c06-9e22-57617168eb95-kube-api-access-px25s\") pod \"bca93dd0-4a59-4c06-9e22-57617168eb95\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") "
Sep 16 12:37:26.676069 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.675814    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-model-cache\") pod \"bca93dd0-4a59-4c06-9e22-57617168eb95\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") "
Sep 16 12:37:26.676069 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.675851    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-home\") pod \"bca93dd0-4a59-4c06-9e22-57617168eb95\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") "
Sep 16 12:37:26.676069 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.675903    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-tmp-dir\") pod \"bca93dd0-4a59-4c06-9e22-57617168eb95\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") "
Sep 16 12:37:26.676069 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.675920    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-dshm\") pod \"bca93dd0-4a59-4c06-9e22-57617168eb95\" (UID: \"bca93dd0-4a59-4c06-9e22-57617168eb95\") "
Sep 16 12:37:26.676265 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.676104    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-home" (OuterVolumeSpecName: "home") pod "bca93dd0-4a59-4c06-9e22-57617168eb95" (UID: "bca93dd0-4a59-4c06-9e22-57617168eb95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:37:26.676265 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.676128    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bca93dd0-4a59-4c06-9e22-57617168eb95" (UID: "bca93dd0-4a59-4c06-9e22-57617168eb95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:37:26.676265 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.676114    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-model-cache" (OuterVolumeSpecName: "model-cache") pod "bca93dd0-4a59-4c06-9e22-57617168eb95" (UID: "bca93dd0-4a59-4c06-9e22-57617168eb95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:37:26.676375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.676285    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-tmp-dir\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:37:26.676375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.676300    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-model-cache\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:37:26.676375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.676309    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-home\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:37:26.678089 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.678052    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-dshm" (OuterVolumeSpecName: "dshm") pod "bca93dd0-4a59-4c06-9e22-57617168eb95" (UID: "bca93dd0-4a59-4c06-9e22-57617168eb95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:37:26.678203 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.678099    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca93dd0-4a59-4c06-9e22-57617168eb95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bca93dd0-4a59-4c06-9e22-57617168eb95" (UID: "bca93dd0-4a59-4c06-9e22-57617168eb95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:37:26.678203 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.678158    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca93dd0-4a59-4c06-9e22-57617168eb95-kube-api-access-px25s" (OuterVolumeSpecName: "kube-api-access-px25s") pod "bca93dd0-4a59-4c06-9e22-57617168eb95" (UID: "bca93dd0-4a59-4c06-9e22-57617168eb95"). InnerVolumeSpecName "kube-api-access-px25s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:37:26.777747 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.777650    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca93dd0-4a59-4c06-9e22-57617168eb95-dshm\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:37:26.777747 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.777681    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca93dd0-4a59-4c06-9e22-57617168eb95-tls-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:37:26.777747 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.777693    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px25s\" (UniqueName: \"kubernetes.io/projected/bca93dd0-4a59-4c06-9e22-57617168eb95-kube-api-access-px25s\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:37:26.940139 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.940105    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"]
Sep 16 12:37:26.944028 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:26.943998    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/rate-test-model-0a9008-kserve-54479c799f-zshh4"]
Sep 16 12:37:27.603736 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:27.603708    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:37:27.770642 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:27.768938    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bca93dd0-4a59-4c06-9e22-57617168eb95" path="/var/lib/kubelet/pods/bca93dd0-4a59-4c06-9e22-57617168eb95/volumes"
Sep 16 12:37:31.595534 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:31.595503    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"
Sep 16 12:37:31.599700 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:31.599663    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 12:37:31.599845 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:31.599712    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 12:37:33.657032 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:33.656997    2558 generic.go:356] "Generic (PLEG): container finished" podID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerID="d89d2ad520e77bd30653c3bced80c6b66fd3f5f7e2e990f78407d3d0f2156dcd" exitCode=0
Sep 16 12:37:33.657480 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:33.657048    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" event={"ID":"a01c434a-188c-4cc4-a835-1fbf833440d5","Type":"ContainerDied","Data":"d89d2ad520e77bd30653c3bced80c6b66fd3f5f7e2e990f78407d3d0f2156dcd"}
Sep 16 12:37:33.657526 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:33.657501    2558 scope.go:117] "RemoveContainer" containerID="d89d2ad520e77bd30653c3bced80c6b66fd3f5f7e2e990f78407d3d0f2156dcd"
Sep 16 12:37:33.658579 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:33.658525    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:37:33.658692 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:33.658639    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:37:33.658692 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:33.658672    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:37:34.668639 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:34.668602    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" event={"ID":"a01c434a-188c-4cc4-a835-1fbf833440d5","Type":"ContainerStarted","Data":"caba0a2750fd84d75c75725171ca3c0cd43c4b6d1a22319441bb7c8ca7ee7550"}
Sep 16 12:37:35.347775 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:35.347730    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:35.348077 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:35.348038    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 12:37:35.348162 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:35.348111    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 12:37:36.348030 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:36.347980    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 12:37:36.348565 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:36.348054    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 12:37:37.347824 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:37.347777    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 12:37:37.348049 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:37.347853    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 12:37:38.348118 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:38.348079    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 12:37:38.348646 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:38.348156    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 12:37:39.348525 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:39.348481    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 12:37:39.349059 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:39.348581    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 12:37:39.987450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:39.987407    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:40.351474 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:40.351443    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:40.351871 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:40.351751    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:40.352390 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:40.352373    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:37:41.514271 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:41.514243    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:41.614438 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:41.614407    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:42.250499 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.250458    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:42.349249 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.349214    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:42.526523 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.526439    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"]
Sep 16 12:37:42.526895 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.526881    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca93dd0-4a59-4c06-9e22-57617168eb95" containerName="main"
Sep 16 12:37:42.526943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.526897    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca93dd0-4a59-4c06-9e22-57617168eb95" containerName="main"
Sep 16 12:37:42.526981 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.526956    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca93dd0-4a59-4c06-9e22-57617168eb95" containerName="main"
Sep 16 12:37:42.530261 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.530236    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.533009 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.532984    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-89b7ca-openshift-default-dockercfg-4jv7p\""
Sep 16 12:37:42.546313 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.546287    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"]
Sep 16 12:37:42.582049 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.582018    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:42.627703 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.627668    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjx6\" (UniqueName: \"kubernetes.io/projected/af634e00-dcc3-4748-9e01-2fd6675f622f-kube-api-access-8kjx6\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.627911 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.627710    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-workload-socket\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.627911 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.627834    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-credential-socket\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.627911 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.627878    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af634e00-dcc3-4748-9e01-2fd6675f622f-istiod-ca-cert\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.628159 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.627980    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-workload-certs\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.628159 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.628012    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-data\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.628159 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.628029    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-podinfo\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.628159 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.628060    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-envoy\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.628380 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.628157    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-token\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.729149 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.729109    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjx6\" (UniqueName: \"kubernetes.io/projected/af634e00-dcc3-4748-9e01-2fd6675f622f-kube-api-access-8kjx6\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.729149 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.729148    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-workload-socket\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.729422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.729179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-credential-socket\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.729422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.729197    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af634e00-dcc3-4748-9e01-2fd6675f622f-istiod-ca-cert\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.729422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.729234    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-workload-certs\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.729422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.729249    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-data\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.729692 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.729499    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-podinfo\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.729692 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.729561    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-envoy\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.729692 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.729618    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-token\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.729692 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.729629    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-workload-socket\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.729906 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.729747    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-credential-socket\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.729906 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.729807    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-workload-certs\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.730111 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.730084    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af634e00-dcc3-4748-9e01-2fd6675f622f-istiod-ca-cert\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.730234 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.730086    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-data\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.732022 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.732000    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-envoy\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.732163 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.732144    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-podinfo\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.740083 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.740054    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-token\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.740228 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.740207    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjx6\" (UniqueName: \"kubernetes.io/projected/af634e00-dcc3-4748-9e01-2fd6675f622f-kube-api-access-8kjx6\") pod \"e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") " pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:42.841207 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:42.841178    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:43.129045 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:43.129012    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"]
Sep 16 12:37:43.131186 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:37:43.131155    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf634e00_dcc3_4748_9e01_2fd6675f622f.slice/crio-3ebccb6b465ae5acdea84a182163ceb1bd547113161ec70285bcdf75c3850b27 WatchSource:0}: Error finding container 3ebccb6b465ae5acdea84a182163ceb1bd547113161ec70285bcdf75c3850b27: Status 404 returned error can't find the container with id 3ebccb6b465ae5acdea84a182163ceb1bd547113161ec70285bcdf75c3850b27
Sep 16 12:37:43.133440 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:43.133402    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:37:43.133525 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:43.133476    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:37:43.133525 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:43.133504    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:37:43.246213 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:43.246177    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:43.344319 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:43.344285    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:43.715837 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:43.715796    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" event={"ID":"af634e00-dcc3-4748-9e01-2fd6675f622f","Type":"ContainerStarted","Data":"c2283f137bb2362e1afffc413eb9885659a28665d58335f93a5345f883b5efb6"}
Sep 16 12:37:43.716257 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:43.715864    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" event={"ID":"af634e00-dcc3-4748-9e01-2fd6675f622f","Type":"ContainerStarted","Data":"3ebccb6b465ae5acdea84a182163ceb1bd547113161ec70285bcdf75c3850b27"}
Sep 16 12:37:43.738662 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:43.738612    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" podStartSLOduration=1.738594264 podStartE2EDuration="1.738594264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:37:43.736668507 +0000 UTC m=+2060.490627239" watchObservedRunningTime="2026-09-16 12:37:43.738594264 +0000 UTC m=+2060.492552994"
Sep 16 12:37:43.841776 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:43.841735    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:44.018779 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:44.018689    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:44.847584 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:44.847537    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:45.079372 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:45.079324    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:45.726240 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:45.726204    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:45.727145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:45.727120    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:37:45.753167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:45.753132    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:46.020872 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:46.020720    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:46.160110 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:46.160073    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:47.085041 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:47.084976    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:48.055725 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:48.055680    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:48.879229 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:48.879196    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:48.973433 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:48.973378    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:49.074949 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:49.074912    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:49.522159 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:49.522126    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:49.679566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:37:49.679500    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:01.767745 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:01.767704    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 12:38:01.768257 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:01.767778    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 12:38:03.821704 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:03.821590    2558 generic.go:356] "Generic (PLEG): container finished" podID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerID="c2283f137bb2362e1afffc413eb9885659a28665d58335f93a5345f883b5efb6" exitCode=0
Sep 16 12:38:03.821704 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:03.821669    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" event={"ID":"af634e00-dcc3-4748-9e01-2fd6675f622f","Type":"ContainerDied","Data":"c2283f137bb2362e1afffc413eb9885659a28665d58335f93a5345f883b5efb6"}
Sep 16 12:38:03.822173 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:03.822160    2558 scope.go:117] "RemoveContainer" containerID="c2283f137bb2362e1afffc413eb9885659a28665d58335f93a5345f883b5efb6"
Sep 16 12:38:03.823320 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:03.823289    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:38:03.823424 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:03.823358    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:38:03.823424 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:03.823389    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:38:04.829199 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:04.829160    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" event={"ID":"af634e00-dcc3-4748-9e01-2fd6675f622f","Type":"ContainerStarted","Data":"95d75f0a15fd14dcffaa5f9df7cbaf02cfee031b1e824b81ab7791dc85a0e513"}
Sep 16 12:38:04.841709 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:04.841674    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:38:04.841936 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:04.841913    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 12:38:04.842002 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:04.841963    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 12:38:05.842149 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:05.842111    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 12:38:05.842533 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:05.842171    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 12:38:06.841869 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:06.841832    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 12:38:06.842059 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:06.841895    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 12:38:07.842023 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:07.841978    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 12:38:07.842439 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:07.842058    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 12:38:08.842450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:08.842412    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 12:38:08.842959 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:08.842480    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 12:38:09.850947 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:09.850910    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:38:09.851519 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:09.851413    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:38:09.852387 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:09.852351    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:38:11.153279 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:11.153249    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:11.846819 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:11.846784    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:12.720983 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:12.720945    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:13.394054 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:13.394021    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:14.237692 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:14.237658    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:15.820114 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:15.820080    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:21.211319 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:21.211284    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:28.664332 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:28.664294    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:49.825251 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:49.825217    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:56.899777 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:56.899741    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz_baf7908c-8e7c-4c5a-802b-a6bceb64075a/payload-processing/0.log"
Sep 16 12:38:57.425822 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.425784    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:57.722897 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.722815    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"]
Sep 16 12:38:57.726448 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.726431    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.729287 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.729268    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-b03a3f-openshift-default-dockercfg-586lg\""
Sep 16 12:38:57.742161 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.742133    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:57.745280 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.745258    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"]
Sep 16 12:38:57.875800 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.875766    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-workload-certs\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.875996 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.875811    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngjwn\" (UniqueName: \"kubernetes.io/projected/ffade955-1f51-4d20-9d4f-cedb751cf42b-kube-api-access-ngjwn\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.875996 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.875899    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffade955-1f51-4d20-9d4f-cedb751cf42b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.876100 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.876065    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-credential-socket\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.876193 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.876172    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-podinfo\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.876258 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.876222    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-envoy\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.876258 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.876250    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-data\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.876348 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.876333    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-workload-socket\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.876384 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.876369    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-token\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.977787 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.977678    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-envoy\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.977787 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.977725    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-data\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.977787 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.977786    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-workload-socket\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.978371 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.977823    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-token\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.978371 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.977870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-workload-certs\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.978371 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.977906    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngjwn\" (UniqueName: \"kubernetes.io/projected/ffade955-1f51-4d20-9d4f-cedb751cf42b-kube-api-access-ngjwn\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.978371 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.977953    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffade955-1f51-4d20-9d4f-cedb751cf42b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.978371 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.977989    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-credential-socket\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.978371 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.978055    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-podinfo\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.978371 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.978205    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-data\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.978371 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.978239    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-workload-certs\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.978759 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.978415    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-credential-socket\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.978759 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.978478    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-workload-socket\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.978759 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.978732    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffade955-1f51-4d20-9d4f-cedb751cf42b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.980109 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.980090    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-envoy\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.980454 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.980429    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-podinfo\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.985987 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.985959    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-token\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:57.986384 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:57.986364    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngjwn\" (UniqueName: \"kubernetes.io/projected/ffade955-1f51-4d20-9d4f-cedb751cf42b-kube-api-access-ngjwn\") pod \"e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") " pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:58.037123 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:58.037086    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:38:58.324365 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:58.324253    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"]
Sep 16 12:38:58.326710 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:38:58.326681    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffade955_1f51_4d20_9d4f_cedb751cf42b.slice/crio-bc6bd3288a9cc08807cc6cf6871623fcd9b3a38e976bead1f251199ec3fd5ce9 WatchSource:0}: Error finding container bc6bd3288a9cc08807cc6cf6871623fcd9b3a38e976bead1f251199ec3fd5ce9: Status 404 returned error can't find the container with id bc6bd3288a9cc08807cc6cf6871623fcd9b3a38e976bead1f251199ec3fd5ce9
Sep 16 12:38:58.328983 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:58.328949    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:38:58.329056 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:58.329017    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:38:58.329056 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:58.329048    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:38:58.681271 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:58.681242    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:59.111379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:59.111339    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79" event={"ID":"ffade955-1f51-4d20-9d4f-cedb751cf42b","Type":"ContainerStarted","Data":"96a6d33b8b6bb98c21991fe36fbf80cff45c5682e641ad1379e7bdc924d464f7"}
Sep 16 12:38:59.111949 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:59.111418    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79" event={"ID":"ffade955-1f51-4d20-9d4f-cedb751cf42b","Type":"ContainerStarted","Data":"bc6bd3288a9cc08807cc6cf6871623fcd9b3a38e976bead1f251199ec3fd5ce9"}
Sep 16 12:38:59.137212 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:38:59.137158    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79" podStartSLOduration=2.137142908 podStartE2EDuration="2.137142908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:59.13574809 +0000 UTC m=+2135.889706821" watchObservedRunningTime="2026-09-16 12:38:59.137142908 +0000 UTC m=+2135.891101638"
Sep 16 12:39:00.037441 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:00.037404    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:39:00.043091 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:00.043058    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:39:00.116961 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:00.116922    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:39:00.118024 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:00.117998    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:39:00.451694 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:00.451662    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:01.072231 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:01.072194    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:01.204174 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:01.204141    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:01.357090 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:01.357058    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:02.518187 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:02.518145    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:03.382012 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:03.381977    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:04.131499 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:04.131452    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:04.856882 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:04.856850    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:04.932299 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:04.932228    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:05.027805 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.027771    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:05.444005 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.443970    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:05.544017 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.543988    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:05.560381 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.560347    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"]
Sep 16 12:39:05.564093 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.564069    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"
Sep 16 12:39:05.566749 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.566723    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-b03a3f\""
Sep 16 12:39:05.566749 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.566723    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-b03a3f-dockercfg-92ncs\""
Sep 16 12:39:05.582203 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.582171    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"]
Sep 16 12:39:05.642665 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.642625    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b33817-756f-44a7-8f63-6e1d2d07be86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh\" (UID: \"22b33817-756f-44a7-8f63-6e1d2d07be86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"
Sep 16 12:39:05.642842 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.642691    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld4kn\" (UniqueName: \"kubernetes.io/projected/22b33817-756f-44a7-8f63-6e1d2d07be86-kube-api-access-ld4kn\") pod \"payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh\" (UID: \"22b33817-756f-44a7-8f63-6e1d2d07be86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"
Sep 16 12:39:05.743341 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.743235    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld4kn\" (UniqueName: \"kubernetes.io/projected/22b33817-756f-44a7-8f63-6e1d2d07be86-kube-api-access-ld4kn\") pod \"payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh\" (UID: \"22b33817-756f-44a7-8f63-6e1d2d07be86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"
Sep 16 12:39:05.743341 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.743334    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b33817-756f-44a7-8f63-6e1d2d07be86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh\" (UID: \"22b33817-756f-44a7-8f63-6e1d2d07be86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"
Sep 16 12:39:05.744026 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.744005    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b33817-756f-44a7-8f63-6e1d2d07be86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh\" (UID: \"22b33817-756f-44a7-8f63-6e1d2d07be86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"
Sep 16 12:39:05.751493 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.751464    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld4kn\" (UniqueName: \"kubernetes.io/projected/22b33817-756f-44a7-8f63-6e1d2d07be86-kube-api-access-ld4kn\") pod \"payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh\" (UID: \"22b33817-756f-44a7-8f63-6e1d2d07be86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"
Sep 16 12:39:05.875563 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:05.875521    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"
Sep 16 12:39:06.155663 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:06.155531    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"]
Sep 16 12:39:06.158118 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:39:06.158087    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b33817_756f_44a7_8f63_6e1d2d07be86.slice/crio-bd402784debf2f17a6f7cc4eb50fca14fba93bc7f3e83e16813665db20b03f19 WatchSource:0}: Error finding container bd402784debf2f17a6f7cc4eb50fca14fba93bc7f3e83e16813665db20b03f19: Status 404 returned error can't find the container with id bd402784debf2f17a6f7cc4eb50fca14fba93bc7f3e83e16813665db20b03f19
Sep 16 12:39:07.155671 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:07.155634    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh" event={"ID":"22b33817-756f-44a7-8f63-6e1d2d07be86","Type":"ContainerStarted","Data":"acb4b7637c601acae8d265859a7bf368455b7703392a9206e44af2257511fb21"}
Sep 16 12:39:07.156127 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:07.155685    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh" event={"ID":"22b33817-756f-44a7-8f63-6e1d2d07be86","Type":"ContainerStarted","Data":"bd402784debf2f17a6f7cc4eb50fca14fba93bc7f3e83e16813665db20b03f19"}
Sep 16 12:39:07.156127 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:07.155717    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"
Sep 16 12:39:07.173379 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:07.173326    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh" podStartSLOduration=2.1733089740000002 podStartE2EDuration="2.173308974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:39:07.17177366 +0000 UTC m=+2143.925732393" watchObservedRunningTime="2026-09-16 12:39:07.173308974 +0000 UTC m=+2143.927267704"
Sep 16 12:39:08.946790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:08.946760    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:18.163575 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:18.163467    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"
Sep 16 12:39:22.236222 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:22.236185    2558 generic.go:356] "Generic (PLEG): container finished" podID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerID="96a6d33b8b6bb98c21991fe36fbf80cff45c5682e641ad1379e7bdc924d464f7" exitCode=0
Sep 16 12:39:22.236789 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:22.236263    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79" event={"ID":"ffade955-1f51-4d20-9d4f-cedb751cf42b","Type":"ContainerDied","Data":"96a6d33b8b6bb98c21991fe36fbf80cff45c5682e641ad1379e7bdc924d464f7"}
Sep 16 12:39:22.236857 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:22.236841    2558 scope.go:117] "RemoveContainer" containerID="96a6d33b8b6bb98c21991fe36fbf80cff45c5682e641ad1379e7bdc924d464f7"
Sep 16 12:39:22.238038 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:22.238009    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:39:22.238145 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:22.238090    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:39:22.238207 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:22.238138    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:39:23.243846 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:23.243813    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79" event={"ID":"ffade955-1f51-4d20-9d4f-cedb751cf42b","Type":"ContainerStarted","Data":"3ac68a951e905aa98bc4e2ebaf6d6e1b12788778aee5026873b72b5cc6cd37ec"}
Sep 16 12:39:24.037301 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:24.037257    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:39:24.037539 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:24.037510    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 16 12:39:24.037623 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:24.037601    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79" podUID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 16 12:39:25.038227 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:25.038191    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 16 12:39:25.038670 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:25.038251    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79" podUID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 16 12:39:26.025698 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:26.025664    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:26.037632 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:26.037602    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 16 12:39:26.037808 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:26.037656    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79" podUID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 16 12:39:27.038345 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:27.038309    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 16 12:39:27.038767 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:27.038370    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79" podUID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 16 12:39:27.559412 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:27.559340    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:28.038297 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:28.038265    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 16 12:39:28.038582 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:28.038535    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79" podUID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 16 12:39:28.372360 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:28.372303    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:29.042667 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:29.042635    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:39:29.043076 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:29.042900    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:39:29.043673 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:29.043654    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:39:29.234579 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:29.234533    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:29.984654 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:29.984620    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:30.682086 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:30.682055    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:31.255399 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:31.255359    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:31.530468 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:31.530358    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:32.543785 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:32.543755    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:33.330874 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:33.330833    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:34.165781 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:34.165721    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:34.971623 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:34.971573    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:35.781470 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:35.781429    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:36.646838 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:36.646803    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:42.957687 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:42.957641    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:43.095935 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.095872    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:43.144779 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.144742    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"]
Sep 16 12:39:43.145021 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.144994    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh" podUID="22b33817-756f-44a7-8f63-6e1d2d07be86" containerName="payload-pre-processing" containerID="cri-o://acb4b7637c601acae8d265859a7bf368455b7703392a9206e44af2257511fb21" gracePeriod=30
Sep 16 12:39:43.196437 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.196409    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:43.306482 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.306446    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:43.364758 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.364710    2558 generic.go:356] "Generic (PLEG): container finished" podID="22b33817-756f-44a7-8f63-6e1d2d07be86" containerID="acb4b7637c601acae8d265859a7bf368455b7703392a9206e44af2257511fb21" exitCode=0
Sep 16 12:39:43.364925 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.364865    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh" event={"ID":"22b33817-756f-44a7-8f63-6e1d2d07be86","Type":"ContainerDied","Data":"acb4b7637c601acae8d265859a7bf368455b7703392a9206e44af2257511fb21"}
Sep 16 12:39:43.401061 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.401028    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:43.420529 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.420492    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"
Sep 16 12:39:43.526088 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.525986    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b33817-756f-44a7-8f63-6e1d2d07be86-plugins-config-volume\") pod \"22b33817-756f-44a7-8f63-6e1d2d07be86\" (UID: \"22b33817-756f-44a7-8f63-6e1d2d07be86\") "
Sep 16 12:39:43.526249 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.526127    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld4kn\" (UniqueName: \"kubernetes.io/projected/22b33817-756f-44a7-8f63-6e1d2d07be86-kube-api-access-ld4kn\") pod \"22b33817-756f-44a7-8f63-6e1d2d07be86\" (UID: \"22b33817-756f-44a7-8f63-6e1d2d07be86\") "
Sep 16 12:39:43.526467 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.526440    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b33817-756f-44a7-8f63-6e1d2d07be86-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22b33817-756f-44a7-8f63-6e1d2d07be86" (UID: "22b33817-756f-44a7-8f63-6e1d2d07be86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:39:43.528264 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.528234    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b33817-756f-44a7-8f63-6e1d2d07be86-kube-api-access-ld4kn" (OuterVolumeSpecName: "kube-api-access-ld4kn") pod "22b33817-756f-44a7-8f63-6e1d2d07be86" (UID: "22b33817-756f-44a7-8f63-6e1d2d07be86"). InnerVolumeSpecName "kube-api-access-ld4kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:39:43.627157 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.627118    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b33817-756f-44a7-8f63-6e1d2d07be86-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:39:43.627157 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.627149    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld4kn\" (UniqueName: \"kubernetes.io/projected/22b33817-756f-44a7-8f63-6e1d2d07be86-kube-api-access-ld4kn\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:39:43.745233 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.745196    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:43.860819 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:43.860784    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:44.371405 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:44.371373    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"
Sep 16 12:39:44.371891 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:44.371408    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh" event={"ID":"22b33817-756f-44a7-8f63-6e1d2d07be86","Type":"ContainerDied","Data":"bd402784debf2f17a6f7cc4eb50fca14fba93bc7f3e83e16813665db20b03f19"}
Sep 16 12:39:44.371891 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:44.371479    2558 scope.go:117] "RemoveContainer" containerID="acb4b7637c601acae8d265859a7bf368455b7703392a9206e44af2257511fb21"
Sep 16 12:39:44.392773 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:44.392735    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"]
Sep 16 12:39:44.401579 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:44.400136    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b03a3f-5694f9cfcd-h29zh"]
Sep 16 12:39:44.602714 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:44.602679    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:45.377601 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:45.377539    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:45.759133 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:45.759045    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22b33817-756f-44a7-8f63-6e1d2d07be86" path="/var/lib/kubelet/pods/22b33817-756f-44a7-8f63-6e1d2d07be86/volumes"
Sep 16 12:39:46.156987 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:46.156947    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:46.835102 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:46.835068    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:49.786846 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:49.786805    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:49.862167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:49.862135    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:50.036532 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:50.036498    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"]
Sep 16 12:39:50.036806 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:50.036780    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79" podUID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerName="istio-proxy" containerID="cri-o://3ac68a951e905aa98bc4e2ebaf6d6e1b12788778aee5026873b72b5cc6cd37ec" gracePeriod=30
Sep 16 12:39:50.063917 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:50.063886    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:54.252736 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:54.252679    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:55.029498 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.029451    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:55.298961 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.298934    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:39:55.338514 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.338483    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-workload-certs\") pod \"ffade955-1f51-4d20-9d4f-cedb751cf42b\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") "
Sep 16 12:39:55.338724 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.338528    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-credential-socket\") pod \"ffade955-1f51-4d20-9d4f-cedb751cf42b\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") "
Sep 16 12:39:55.338724 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.338582    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffade955-1f51-4d20-9d4f-cedb751cf42b-istiod-ca-cert\") pod \"ffade955-1f51-4d20-9d4f-cedb751cf42b\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") "
Sep 16 12:39:55.338724 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.338612    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-workload-socket\") pod \"ffade955-1f51-4d20-9d4f-cedb751cf42b\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") "
Sep 16 12:39:55.338724 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.338638    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngjwn\" (UniqueName: \"kubernetes.io/projected/ffade955-1f51-4d20-9d4f-cedb751cf42b-kube-api-access-ngjwn\") pod \"ffade955-1f51-4d20-9d4f-cedb751cf42b\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") "
Sep 16 12:39:55.338724 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.338685    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-envoy\") pod \"ffade955-1f51-4d20-9d4f-cedb751cf42b\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") "
Sep 16 12:39:55.338997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.338726    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-podinfo\") pod \"ffade955-1f51-4d20-9d4f-cedb751cf42b\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") "
Sep 16 12:39:55.338997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.338753    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-token\") pod \"ffade955-1f51-4d20-9d4f-cedb751cf42b\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") "
Sep 16 12:39:55.338997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.338785    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-data\") pod \"ffade955-1f51-4d20-9d4f-cedb751cf42b\" (UID: \"ffade955-1f51-4d20-9d4f-cedb751cf42b\") "
Sep 16 12:39:55.338997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.338807    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ffade955-1f51-4d20-9d4f-cedb751cf42b" (UID: "ffade955-1f51-4d20-9d4f-cedb751cf42b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:39:55.338997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.338942    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ffade955-1f51-4d20-9d4f-cedb751cf42b" (UID: "ffade955-1f51-4d20-9d4f-cedb751cf42b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:39:55.338997 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.338952    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ffade955-1f51-4d20-9d4f-cedb751cf42b" (UID: "ffade955-1f51-4d20-9d4f-cedb751cf42b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:39:55.339303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.339033    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffade955-1f51-4d20-9d4f-cedb751cf42b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ffade955-1f51-4d20-9d4f-cedb751cf42b" (UID: "ffade955-1f51-4d20-9d4f-cedb751cf42b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:39:55.339303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.339160    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-data" (OuterVolumeSpecName: "istio-data") pod "ffade955-1f51-4d20-9d4f-cedb751cf42b" (UID: "ffade955-1f51-4d20-9d4f-cedb751cf42b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:39:55.339303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.339184    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-workload-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:39:55.339303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.339205    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-credential-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:39:55.339303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.339220    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffade955-1f51-4d20-9d4f-cedb751cf42b-istiod-ca-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:39:55.339303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.339234    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-workload-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:39:55.341214 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.341179    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ffade955-1f51-4d20-9d4f-cedb751cf42b" (UID: "ffade955-1f51-4d20-9d4f-cedb751cf42b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:39:55.341214 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.341193    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ffade955-1f51-4d20-9d4f-cedb751cf42b" (UID: "ffade955-1f51-4d20-9d4f-cedb751cf42b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:39:55.341414 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.341384    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffade955-1f51-4d20-9d4f-cedb751cf42b-kube-api-access-ngjwn" (OuterVolumeSpecName: "kube-api-access-ngjwn") pod "ffade955-1f51-4d20-9d4f-cedb751cf42b" (UID: "ffade955-1f51-4d20-9d4f-cedb751cf42b"). InnerVolumeSpecName "kube-api-access-ngjwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:39:55.341521 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.341502    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-token" (OuterVolumeSpecName: "istio-token") pod "ffade955-1f51-4d20-9d4f-cedb751cf42b" (UID: "ffade955-1f51-4d20-9d4f-cedb751cf42b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:39:55.430381 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.430346    2558 generic.go:356] "Generic (PLEG): container finished" podID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerID="3ac68a951e905aa98bc4e2ebaf6d6e1b12788778aee5026873b72b5cc6cd37ec" exitCode=0
Sep 16 12:39:55.430569 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.430392    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"
Sep 16 12:39:55.430569 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.430411    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79" event={"ID":"ffade955-1f51-4d20-9d4f-cedb751cf42b","Type":"ContainerDied","Data":"3ac68a951e905aa98bc4e2ebaf6d6e1b12788778aee5026873b72b5cc6cd37ec"}
Sep 16 12:39:55.430569 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.430454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79" event={"ID":"ffade955-1f51-4d20-9d4f-cedb751cf42b","Type":"ContainerDied","Data":"bc6bd3288a9cc08807cc6cf6871623fcd9b3a38e976bead1f251199ec3fd5ce9"}
Sep 16 12:39:55.430569 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.430476    2558 scope.go:117] "RemoveContainer" containerID="3ac68a951e905aa98bc4e2ebaf6d6e1b12788778aee5026873b72b5cc6cd37ec"
Sep 16 12:39:55.440085 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.440065    2558 scope.go:117] "RemoveContainer" containerID="96a6d33b8b6bb98c21991fe36fbf80cff45c5682e641ad1379e7bdc924d464f7"
Sep 16 12:39:55.440388 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.440365    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngjwn\" (UniqueName: \"kubernetes.io/projected/ffade955-1f51-4d20-9d4f-cedb751cf42b-kube-api-access-ngjwn\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:39:55.440458 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.440392    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-envoy\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:39:55.440458 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.440402    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-podinfo\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:39:55.440458 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.440411    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-token\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:39:55.440458 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.440418    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffade955-1f51-4d20-9d4f-cedb751cf42b-istio-data\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:39:55.448350 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.448325    2558 scope.go:117] "RemoveContainer" containerID="3ac68a951e905aa98bc4e2ebaf6d6e1b12788778aee5026873b72b5cc6cd37ec"
Sep 16 12:39:55.448891 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:39:55.448840    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac68a951e905aa98bc4e2ebaf6d6e1b12788778aee5026873b72b5cc6cd37ec\": container with ID starting with 3ac68a951e905aa98bc4e2ebaf6d6e1b12788778aee5026873b72b5cc6cd37ec not found: ID does not exist" containerID="3ac68a951e905aa98bc4e2ebaf6d6e1b12788778aee5026873b72b5cc6cd37ec"
Sep 16 12:39:55.449004 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.448901    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac68a951e905aa98bc4e2ebaf6d6e1b12788778aee5026873b72b5cc6cd37ec"} err="failed to get container status \"3ac68a951e905aa98bc4e2ebaf6d6e1b12788778aee5026873b72b5cc6cd37ec\": rpc error: code = NotFound desc = could not find container \"3ac68a951e905aa98bc4e2ebaf6d6e1b12788778aee5026873b72b5cc6cd37ec\": container with ID starting with 3ac68a951e905aa98bc4e2ebaf6d6e1b12788778aee5026873b72b5cc6cd37ec not found: ID does not exist"
Sep 16 12:39:55.449004 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.448933    2558 scope.go:117] "RemoveContainer" containerID="96a6d33b8b6bb98c21991fe36fbf80cff45c5682e641ad1379e7bdc924d464f7"
Sep 16 12:39:55.449367 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:39:55.449297    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96a6d33b8b6bb98c21991fe36fbf80cff45c5682e641ad1379e7bdc924d464f7\": container with ID starting with 96a6d33b8b6bb98c21991fe36fbf80cff45c5682e641ad1379e7bdc924d464f7 not found: ID does not exist" containerID="96a6d33b8b6bb98c21991fe36fbf80cff45c5682e641ad1379e7bdc924d464f7"
Sep 16 12:39:55.449367 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.449336    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a6d33b8b6bb98c21991fe36fbf80cff45c5682e641ad1379e7bdc924d464f7"} err="failed to get container status \"96a6d33b8b6bb98c21991fe36fbf80cff45c5682e641ad1379e7bdc924d464f7\": rpc error: code = NotFound desc = could not find container \"96a6d33b8b6bb98c21991fe36fbf80cff45c5682e641ad1379e7bdc924d464f7\": container with ID starting with 96a6d33b8b6bb98c21991fe36fbf80cff45c5682e641ad1379e7bdc924d464f7 not found: ID does not exist"
Sep 16 12:39:55.451418 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.451399    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"]
Sep 16 12:39:55.457556 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.457522    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b03a3f-openshift-default-5ff98cd574-7sb79"]
Sep 16 12:39:55.711495 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.711463    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:55.758099 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:55.758069    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffade955-1f51-4d20-9d4f-cedb751cf42b" path="/var/lib/kubelet/pods/ffade955-1f51-4d20-9d4f-cedb751cf42b/volumes"
Sep 16 12:39:56.517263 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:56.517228    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:57.195247 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:57.195216    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:57.878450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:39:57.878413    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:09.568685 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.568649    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"]
Sep 16 12:40:09.569176 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.568860    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d" podUID="3a1da320-b39d-428f-a1ea-385d9bd1577a" containerName="maas-api" containerID="cri-o://2764ecbf79c3a2e4d67044ad03c1591a43fb12e7b24c7796a777e3f370a97c36" gracePeriod=30
Sep 16 12:40:09.605509 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.605477    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:09.634403 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.634363    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"]
Sep 16 12:40:09.634743 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.634713    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz" podUID="baf7908c-8e7c-4c5a-802b-a6bceb64075a" containerName="payload-processing" containerID="cri-o://fb772d6d9a5b7e475d6e918eae54dae68557949a9441e83b67e764a2b9139b6a" gracePeriod=30
Sep 16 12:40:09.705620 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.705569    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:09.801834 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.801764    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:09.867739 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.867573    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:09.893589 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.893539    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:40:09.934084 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.934055    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"
Sep 16 12:40:09.982762 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.982726    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1da320-b39d-428f-a1ea-385d9bd1577a-openshift-service-ca\") pod \"3a1da320-b39d-428f-a1ea-385d9bd1577a\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") "
Sep 16 12:40:09.982762 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.982765    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-maas-api-tls\") pod \"3a1da320-b39d-428f-a1ea-385d9bd1577a\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") "
Sep 16 12:40:09.982983 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.982804    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-metrics-certs\") pod \"3a1da320-b39d-428f-a1ea-385d9bd1577a\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") "
Sep 16 12:40:09.982983 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.982884    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s5rp\" (UniqueName: \"kubernetes.io/projected/3a1da320-b39d-428f-a1ea-385d9bd1577a-kube-api-access-5s5rp\") pod \"3a1da320-b39d-428f-a1ea-385d9bd1577a\" (UID: \"3a1da320-b39d-428f-a1ea-385d9bd1577a\") "
Sep 16 12:40:09.983302 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.983273    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1da320-b39d-428f-a1ea-385d9bd1577a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3a1da320-b39d-428f-a1ea-385d9bd1577a" (UID: "3a1da320-b39d-428f-a1ea-385d9bd1577a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:40:09.984929 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.984910    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1da320-b39d-428f-a1ea-385d9bd1577a-kube-api-access-5s5rp" (OuterVolumeSpecName: "kube-api-access-5s5rp") pod "3a1da320-b39d-428f-a1ea-385d9bd1577a" (UID: "3a1da320-b39d-428f-a1ea-385d9bd1577a"). InnerVolumeSpecName "kube-api-access-5s5rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:40:09.985327 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.985310    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3a1da320-b39d-428f-a1ea-385d9bd1577a" (UID: "3a1da320-b39d-428f-a1ea-385d9bd1577a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:40:09.985391 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:09.985335    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3a1da320-b39d-428f-a1ea-385d9bd1577a" (UID: "3a1da320-b39d-428f-a1ea-385d9bd1577a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:40:10.084377 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.084278    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf7908c-8e7c-4c5a-802b-a6bceb64075a-plugins-config-volume\") pod \"baf7908c-8e7c-4c5a-802b-a6bceb64075a\" (UID: \"baf7908c-8e7c-4c5a-802b-a6bceb64075a\") "
Sep 16 12:40:10.084572 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.084388    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs4xd\" (UniqueName: \"kubernetes.io/projected/baf7908c-8e7c-4c5a-802b-a6bceb64075a-kube-api-access-fs4xd\") pod \"baf7908c-8e7c-4c5a-802b-a6bceb64075a\" (UID: \"baf7908c-8e7c-4c5a-802b-a6bceb64075a\") "
Sep 16 12:40:10.084632 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.084579    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1da320-b39d-428f-a1ea-385d9bd1577a-openshift-service-ca\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:40:10.084632 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.084591    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-maas-api-tls\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:40:10.084632 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.084600    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1da320-b39d-428f-a1ea-385d9bd1577a-metrics-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:40:10.084632 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.084608    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s5rp\" (UniqueName: \"kubernetes.io/projected/3a1da320-b39d-428f-a1ea-385d9bd1577a-kube-api-access-5s5rp\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:40:10.084787 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.084757    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf7908c-8e7c-4c5a-802b-a6bceb64075a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "baf7908c-8e7c-4c5a-802b-a6bceb64075a" (UID: "baf7908c-8e7c-4c5a-802b-a6bceb64075a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:40:10.086467 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.086445    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf7908c-8e7c-4c5a-802b-a6bceb64075a-kube-api-access-fs4xd" (OuterVolumeSpecName: "kube-api-access-fs4xd") pod "baf7908c-8e7c-4c5a-802b-a6bceb64075a" (UID: "baf7908c-8e7c-4c5a-802b-a6bceb64075a"). InnerVolumeSpecName "kube-api-access-fs4xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:40:10.186154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.186116    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf7908c-8e7c-4c5a-802b-a6bceb64075a-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:40:10.186154 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.186148    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs4xd\" (UniqueName: \"kubernetes.io/projected/baf7908c-8e7c-4c5a-802b-a6bceb64075a-kube-api-access-fs4xd\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:40:10.189906 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.189874    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:10.293230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.293189    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:10.507958 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.507922    2558 generic.go:356] "Generic (PLEG): container finished" podID="3a1da320-b39d-428f-a1ea-385d9bd1577a" containerID="2764ecbf79c3a2e4d67044ad03c1591a43fb12e7b24c7796a777e3f370a97c36" exitCode=0
Sep 16 12:40:10.508167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.507986    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"
Sep 16 12:40:10.508167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.508013    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d" event={"ID":"3a1da320-b39d-428f-a1ea-385d9bd1577a","Type":"ContainerDied","Data":"2764ecbf79c3a2e4d67044ad03c1591a43fb12e7b24c7796a777e3f370a97c36"}
Sep 16 12:40:10.508167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.508111    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d" event={"ID":"3a1da320-b39d-428f-a1ea-385d9bd1577a","Type":"ContainerDied","Data":"3df271ed8d738f05060aae48be4bd82fa4159c6e39b3d89710cdaa842322d92f"}
Sep 16 12:40:10.508167 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.508142    2558 scope.go:117] "RemoveContainer" containerID="2764ecbf79c3a2e4d67044ad03c1591a43fb12e7b24c7796a777e3f370a97c36"
Sep 16 12:40:10.509557 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.509527    2558 generic.go:356] "Generic (PLEG): container finished" podID="baf7908c-8e7c-4c5a-802b-a6bceb64075a" containerID="fb772d6d9a5b7e475d6e918eae54dae68557949a9441e83b67e764a2b9139b6a" exitCode=0
Sep 16 12:40:10.509688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.509578    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz" event={"ID":"baf7908c-8e7c-4c5a-802b-a6bceb64075a","Type":"ContainerDied","Data":"fb772d6d9a5b7e475d6e918eae54dae68557949a9441e83b67e764a2b9139b6a"}
Sep 16 12:40:10.509688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.509608    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"
Sep 16 12:40:10.509688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.509631    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz" event={"ID":"baf7908c-8e7c-4c5a-802b-a6bceb64075a","Type":"ContainerDied","Data":"a889190b0a55b093842f115bd20abfc5ed199d1b6f19a0ac7206fa1237722357"}
Sep 16 12:40:10.520038 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.520019    2558 scope.go:117] "RemoveContainer" containerID="2764ecbf79c3a2e4d67044ad03c1591a43fb12e7b24c7796a777e3f370a97c36"
Sep 16 12:40:10.520409 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:40:10.520297    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2764ecbf79c3a2e4d67044ad03c1591a43fb12e7b24c7796a777e3f370a97c36\": container with ID starting with 2764ecbf79c3a2e4d67044ad03c1591a43fb12e7b24c7796a777e3f370a97c36 not found: ID does not exist" containerID="2764ecbf79c3a2e4d67044ad03c1591a43fb12e7b24c7796a777e3f370a97c36"
Sep 16 12:40:10.520409 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.520327    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2764ecbf79c3a2e4d67044ad03c1591a43fb12e7b24c7796a777e3f370a97c36"} err="failed to get container status \"2764ecbf79c3a2e4d67044ad03c1591a43fb12e7b24c7796a777e3f370a97c36\": rpc error: code = NotFound desc = could not find container \"2764ecbf79c3a2e4d67044ad03c1591a43fb12e7b24c7796a777e3f370a97c36\": container with ID starting with 2764ecbf79c3a2e4d67044ad03c1591a43fb12e7b24c7796a777e3f370a97c36 not found: ID does not exist"
Sep 16 12:40:10.520409 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.520346    2558 scope.go:117] "RemoveContainer" containerID="fb772d6d9a5b7e475d6e918eae54dae68557949a9441e83b67e764a2b9139b6a"
Sep 16 12:40:10.528971 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.528956    2558 scope.go:117] "RemoveContainer" containerID="fb772d6d9a5b7e475d6e918eae54dae68557949a9441e83b67e764a2b9139b6a"
Sep 16 12:40:10.529238 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:40:10.529212    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb772d6d9a5b7e475d6e918eae54dae68557949a9441e83b67e764a2b9139b6a\": container with ID starting with fb772d6d9a5b7e475d6e918eae54dae68557949a9441e83b67e764a2b9139b6a not found: ID does not exist" containerID="fb772d6d9a5b7e475d6e918eae54dae68557949a9441e83b67e764a2b9139b6a"
Sep 16 12:40:10.529331 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.529241    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb772d6d9a5b7e475d6e918eae54dae68557949a9441e83b67e764a2b9139b6a"} err="failed to get container status \"fb772d6d9a5b7e475d6e918eae54dae68557949a9441e83b67e764a2b9139b6a\": rpc error: code = NotFound desc = could not find container \"fb772d6d9a5b7e475d6e918eae54dae68557949a9441e83b67e764a2b9139b6a\": container with ID starting with fb772d6d9a5b7e475d6e918eae54dae68557949a9441e83b67e764a2b9139b6a not found: ID does not exist"
Sep 16 12:40:10.546176 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.546155    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"]
Sep 16 12:40:10.553814 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.553791    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-869204-8ff4b68ff-zm42d"]
Sep 16 12:40:10.566358 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.566335    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"]
Sep 16 12:40:10.577046 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:10.577022    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-869204-78d8fbbc55-4c9zz"]
Sep 16 12:40:11.033617 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:11.033588    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:11.755720 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:11.755689    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:11.758013 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:11.757985    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a1da320-b39d-428f-a1ea-385d9bd1577a" path="/var/lib/kubelet/pods/3a1da320-b39d-428f-a1ea-385d9bd1577a/volumes"
Sep 16 12:40:11.758421 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:11.758408    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf7908c-8e7c-4c5a-802b-a6bceb64075a" path="/var/lib/kubelet/pods/baf7908c-8e7c-4c5a-802b-a6bceb64075a/volumes"
Sep 16 12:40:12.540090 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:12.540056    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:15.263078 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:15.263043    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:25.653871 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:25.653836    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:25.709241 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:25.709208    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:51.840892 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:51.840814    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:53.203465 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:53.203432    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:58.246459 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:58.246426    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:58.326106 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:58.326063    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:58.740532 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:58.740502    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"]
Sep 16 12:40:58.740786 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:58.740746    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerName="istio-proxy" containerID="cri-o://caba0a2750fd84d75c75725171ca3c0cd43c4b6d1a22319441bb7c8ca7ee7550" gracePeriod=30
Sep 16 12:40:58.766761 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:58.766723    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:59.619900 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:40:59.619860    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:00.392327 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:00.392294    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:01.265797 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:01.265760    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:01.936790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:01.936751    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:02.747452 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:02.747419    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:03.988071 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:03.988043    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:41:04.087688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.087586    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhtfp\" (UniqueName: \"kubernetes.io/projected/a01c434a-188c-4cc4-a835-1fbf833440d5-kube-api-access-zhtfp\") pod \"a01c434a-188c-4cc4-a835-1fbf833440d5\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") "
Sep 16 12:41:04.087688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.087650    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-envoy\") pod \"a01c434a-188c-4cc4-a835-1fbf833440d5\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") "
Sep 16 12:41:04.087688 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.087674    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-data\") pod \"a01c434a-188c-4cc4-a835-1fbf833440d5\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") "
Sep 16 12:41:04.087986 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.087696    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-workload-certs\") pod \"a01c434a-188c-4cc4-a835-1fbf833440d5\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") "
Sep 16 12:41:04.087986 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.087732    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01c434a-188c-4cc4-a835-1fbf833440d5-istiod-ca-cert\") pod \"a01c434a-188c-4cc4-a835-1fbf833440d5\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") "
Sep 16 12:41:04.087986 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.087748    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-credential-socket\") pod \"a01c434a-188c-4cc4-a835-1fbf833440d5\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") "
Sep 16 12:41:04.087986 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.087786    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-token\") pod \"a01c434a-188c-4cc4-a835-1fbf833440d5\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") "
Sep 16 12:41:04.087986 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.087808    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-workload-socket\") pod \"a01c434a-188c-4cc4-a835-1fbf833440d5\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") "
Sep 16 12:41:04.087986 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.087871    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-podinfo\") pod \"a01c434a-188c-4cc4-a835-1fbf833440d5\" (UID: \"a01c434a-188c-4cc4-a835-1fbf833440d5\") "
Sep 16 12:41:04.088303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.088026    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-data" (OuterVolumeSpecName: "istio-data") pod "a01c434a-188c-4cc4-a835-1fbf833440d5" (UID: "a01c434a-188c-4cc4-a835-1fbf833440d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:04.088303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.088048    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a01c434a-188c-4cc4-a835-1fbf833440d5" (UID: "a01c434a-188c-4cc4-a835-1fbf833440d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:04.088303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.088222    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-data\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:04.088303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.088243    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-workload-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:04.088303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.088282    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a01c434a-188c-4cc4-a835-1fbf833440d5" (UID: "a01c434a-188c-4cc4-a835-1fbf833440d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:04.088591 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.088320    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01c434a-188c-4cc4-a835-1fbf833440d5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a01c434a-188c-4cc4-a835-1fbf833440d5" (UID: "a01c434a-188c-4cc4-a835-1fbf833440d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:04.088591 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.088484    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a01c434a-188c-4cc4-a835-1fbf833440d5" (UID: "a01c434a-188c-4cc4-a835-1fbf833440d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:04.089940 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.089906    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01c434a-188c-4cc4-a835-1fbf833440d5-kube-api-access-zhtfp" (OuterVolumeSpecName: "kube-api-access-zhtfp") pod "a01c434a-188c-4cc4-a835-1fbf833440d5" (UID: "a01c434a-188c-4cc4-a835-1fbf833440d5"). InnerVolumeSpecName "kube-api-access-zhtfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:04.090081 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.090037    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a01c434a-188c-4cc4-a835-1fbf833440d5" (UID: "a01c434a-188c-4cc4-a835-1fbf833440d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:41:04.090310 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.090294    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "a01c434a-188c-4cc4-a835-1fbf833440d5" (UID: "a01c434a-188c-4cc4-a835-1fbf833440d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:04.090360 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.090301    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a01c434a-188c-4cc4-a835-1fbf833440d5" (UID: "a01c434a-188c-4cc4-a835-1fbf833440d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:04.188796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.188754    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-envoy\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:04.188796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.188789    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01c434a-188c-4cc4-a835-1fbf833440d5-istiod-ca-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:04.188796 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.188802    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-credential-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:04.189046 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.188814    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-token\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:04.189046 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.188825    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c434a-188c-4cc4-a835-1fbf833440d5-workload-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:04.189046 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.188835    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01c434a-188c-4cc4-a835-1fbf833440d5-istio-podinfo\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:04.189046 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.188848    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhtfp\" (UniqueName: \"kubernetes.io/projected/a01c434a-188c-4cc4-a835-1fbf833440d5-kube-api-access-zhtfp\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:04.786083 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.786048    2558 generic.go:356] "Generic (PLEG): container finished" podID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerID="caba0a2750fd84d75c75725171ca3c0cd43c4b6d1a22319441bb7c8ca7ee7550" exitCode=0
Sep 16 12:41:04.786261 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.786115    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"
Sep 16 12:41:04.786261 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.786144    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" event={"ID":"a01c434a-188c-4cc4-a835-1fbf833440d5","Type":"ContainerDied","Data":"caba0a2750fd84d75c75725171ca3c0cd43c4b6d1a22319441bb7c8ca7ee7550"}
Sep 16 12:41:04.786261 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.786205    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq" event={"ID":"a01c434a-188c-4cc4-a835-1fbf833440d5","Type":"ContainerDied","Data":"562da8b9c42ce49926712580d9e2739e32e1b9fe81848ffa0a97d036823c990b"}
Sep 16 12:41:04.786261 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.786230    2558 scope.go:117] "RemoveContainer" containerID="caba0a2750fd84d75c75725171ca3c0cd43c4b6d1a22319441bb7c8ca7ee7550"
Sep 16 12:41:04.795633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.795611    2558 scope.go:117] "RemoveContainer" containerID="d89d2ad520e77bd30653c3bced80c6b66fd3f5f7e2e990f78407d3d0f2156dcd"
Sep 16 12:41:04.803944 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.803922    2558 scope.go:117] "RemoveContainer" containerID="caba0a2750fd84d75c75725171ca3c0cd43c4b6d1a22319441bb7c8ca7ee7550"
Sep 16 12:41:04.804236 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:41:04.804216    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caba0a2750fd84d75c75725171ca3c0cd43c4b6d1a22319441bb7c8ca7ee7550\": container with ID starting with caba0a2750fd84d75c75725171ca3c0cd43c4b6d1a22319441bb7c8ca7ee7550 not found: ID does not exist" containerID="caba0a2750fd84d75c75725171ca3c0cd43c4b6d1a22319441bb7c8ca7ee7550"
Sep 16 12:41:04.804301 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.804246    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caba0a2750fd84d75c75725171ca3c0cd43c4b6d1a22319441bb7c8ca7ee7550"} err="failed to get container status \"caba0a2750fd84d75c75725171ca3c0cd43c4b6d1a22319441bb7c8ca7ee7550\": rpc error: code = NotFound desc = could not find container \"caba0a2750fd84d75c75725171ca3c0cd43c4b6d1a22319441bb7c8ca7ee7550\": container with ID starting with caba0a2750fd84d75c75725171ca3c0cd43c4b6d1a22319441bb7c8ca7ee7550 not found: ID does not exist"
Sep 16 12:41:04.804301 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.804265    2558 scope.go:117] "RemoveContainer" containerID="d89d2ad520e77bd30653c3bced80c6b66fd3f5f7e2e990f78407d3d0f2156dcd"
Sep 16 12:41:04.804501 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:41:04.804486    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89d2ad520e77bd30653c3bced80c6b66fd3f5f7e2e990f78407d3d0f2156dcd\": container with ID starting with d89d2ad520e77bd30653c3bced80c6b66fd3f5f7e2e990f78407d3d0f2156dcd not found: ID does not exist" containerID="d89d2ad520e77bd30653c3bced80c6b66fd3f5f7e2e990f78407d3d0f2156dcd"
Sep 16 12:41:04.804557 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.804503    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89d2ad520e77bd30653c3bced80c6b66fd3f5f7e2e990f78407d3d0f2156dcd"} err="failed to get container status \"d89d2ad520e77bd30653c3bced80c6b66fd3f5f7e2e990f78407d3d0f2156dcd\": rpc error: code = NotFound desc = could not find container \"d89d2ad520e77bd30653c3bced80c6b66fd3f5f7e2e990f78407d3d0f2156dcd\": container with ID starting with d89d2ad520e77bd30653c3bced80c6b66fd3f5f7e2e990f78407d3d0f2156dcd not found: ID does not exist"
Sep 16 12:41:04.810704 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.810681    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"]
Sep 16 12:41:04.818473 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:04.818451    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-869204-openshift-default-d75468dc6-bqkrq"]
Sep 16 12:41:05.757822 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:05.757788    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" path="/var/lib/kubelet/pods/a01c434a-188c-4cc4-a835-1fbf833440d5/volumes"
Sep 16 12:41:10.192470 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:10.192418    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:10.292808 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:10.292764    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:10.444627 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:10.444520    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:10.532017 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:10.531985    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:10.886181 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:10.886136    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:11.006760 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:11.006730    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:11.942346 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:11.942313    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:12.630542 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:12.630511    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:16.756696 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:16.756663    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:16.861006 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:16.860974    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:17.008873 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:17.008789    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"]
Sep 16 12:41:17.009074 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:17.009048    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerName="istio-proxy" containerID="cri-o://95d75f0a15fd14dcffaa5f9df7cbaf02cfee031b1e824b81ab7791dc85a0e513" gracePeriod=30
Sep 16 12:41:17.029777 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:17.029749    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:17.739174 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:17.739136    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:18.670702 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:18.670670    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:19.366408 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:19.366377    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:20.152810 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:20.152779    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:22.271745 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.271675    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:41:22.361157 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361117    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-workload-socket\") pod \"af634e00-dcc3-4748-9e01-2fd6675f622f\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") "
Sep 16 12:41:22.361157 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361164    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-credential-socket\") pod \"af634e00-dcc3-4748-9e01-2fd6675f622f\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") "
Sep 16 12:41:22.361402 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361198    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kjx6\" (UniqueName: \"kubernetes.io/projected/af634e00-dcc3-4748-9e01-2fd6675f622f-kube-api-access-8kjx6\") pod \"af634e00-dcc3-4748-9e01-2fd6675f622f\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") "
Sep 16 12:41:22.361402 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361241    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-workload-certs\") pod \"af634e00-dcc3-4748-9e01-2fd6675f622f\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") "
Sep 16 12:41:22.361402 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361269    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-podinfo\") pod \"af634e00-dcc3-4748-9e01-2fd6675f622f\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") "
Sep 16 12:41:22.361580 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361437    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af634e00-dcc3-4748-9e01-2fd6675f622f-istiod-ca-cert\") pod \"af634e00-dcc3-4748-9e01-2fd6675f622f\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") "
Sep 16 12:41:22.361580 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361445    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af634e00-dcc3-4748-9e01-2fd6675f622f" (UID: "af634e00-dcc3-4748-9e01-2fd6675f622f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:22.361580 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361486    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-envoy\") pod \"af634e00-dcc3-4748-9e01-2fd6675f622f\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") "
Sep 16 12:41:22.361580 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361513    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af634e00-dcc3-4748-9e01-2fd6675f622f" (UID: "af634e00-dcc3-4748-9e01-2fd6675f622f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:22.361799 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361607    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-data\") pod \"af634e00-dcc3-4748-9e01-2fd6675f622f\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") "
Sep 16 12:41:22.361799 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361650    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-token\") pod \"af634e00-dcc3-4748-9e01-2fd6675f622f\" (UID: \"af634e00-dcc3-4748-9e01-2fd6675f622f\") "
Sep 16 12:41:22.361799 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361687    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af634e00-dcc3-4748-9e01-2fd6675f622f" (UID: "af634e00-dcc3-4748-9e01-2fd6675f622f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:22.361941 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361832    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af634e00-dcc3-4748-9e01-2fd6675f622f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af634e00-dcc3-4748-9e01-2fd6675f622f" (UID: "af634e00-dcc3-4748-9e01-2fd6675f622f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:22.361941 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.361856    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-data" (OuterVolumeSpecName: "istio-data") pod "af634e00-dcc3-4748-9e01-2fd6675f622f" (UID: "af634e00-dcc3-4748-9e01-2fd6675f622f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:22.362042 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.362010    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-workload-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:22.362042 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.362030    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af634e00-dcc3-4748-9e01-2fd6675f622f-istiod-ca-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:22.362156 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.362043    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-data\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:22.362156 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.362056    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-workload-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:22.362156 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.362070    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-credential-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:22.363750 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.363726    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af634e00-dcc3-4748-9e01-2fd6675f622f" (UID: "af634e00-dcc3-4748-9e01-2fd6675f622f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:41:22.363750 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.363739    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af634e00-dcc3-4748-9e01-2fd6675f622f-kube-api-access-8kjx6" (OuterVolumeSpecName: "kube-api-access-8kjx6") pod "af634e00-dcc3-4748-9e01-2fd6675f622f" (UID: "af634e00-dcc3-4748-9e01-2fd6675f622f"). InnerVolumeSpecName "kube-api-access-8kjx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:22.363943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.363926    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af634e00-dcc3-4748-9e01-2fd6675f622f" (UID: "af634e00-dcc3-4748-9e01-2fd6675f622f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:22.363991 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.363964    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-token" (OuterVolumeSpecName: "istio-token") pod "af634e00-dcc3-4748-9e01-2fd6675f622f" (UID: "af634e00-dcc3-4748-9e01-2fd6675f622f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:22.462608 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.462541    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-envoy\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:22.462608 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.462601    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-token\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:22.462608 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.462613    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kjx6\" (UniqueName: \"kubernetes.io/projected/af634e00-dcc3-4748-9e01-2fd6675f622f-kube-api-access-8kjx6\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:22.462874 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.462623    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af634e00-dcc3-4748-9e01-2fd6675f622f-istio-podinfo\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:22.882657 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.882620    2558 generic.go:356] "Generic (PLEG): container finished" podID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerID="95d75f0a15fd14dcffaa5f9df7cbaf02cfee031b1e824b81ab7791dc85a0e513" exitCode=0
Sep 16 12:41:22.882840 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.882679    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"
Sep 16 12:41:22.882840 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.882713    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" event={"ID":"af634e00-dcc3-4748-9e01-2fd6675f622f","Type":"ContainerDied","Data":"95d75f0a15fd14dcffaa5f9df7cbaf02cfee031b1e824b81ab7791dc85a0e513"}
Sep 16 12:41:22.882840 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.882772    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t" event={"ID":"af634e00-dcc3-4748-9e01-2fd6675f622f","Type":"ContainerDied","Data":"3ebccb6b465ae5acdea84a182163ceb1bd547113161ec70285bcdf75c3850b27"}
Sep 16 12:41:22.882840 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.882795    2558 scope.go:117] "RemoveContainer" containerID="95d75f0a15fd14dcffaa5f9df7cbaf02cfee031b1e824b81ab7791dc85a0e513"
Sep 16 12:41:22.894802 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.894756    2558 scope.go:117] "RemoveContainer" containerID="c2283f137bb2362e1afffc413eb9885659a28665d58335f93a5345f883b5efb6"
Sep 16 12:41:22.903336 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.903307    2558 scope.go:117] "RemoveContainer" containerID="95d75f0a15fd14dcffaa5f9df7cbaf02cfee031b1e824b81ab7791dc85a0e513"
Sep 16 12:41:22.903867 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:41:22.903834    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d75f0a15fd14dcffaa5f9df7cbaf02cfee031b1e824b81ab7791dc85a0e513\": container with ID starting with 95d75f0a15fd14dcffaa5f9df7cbaf02cfee031b1e824b81ab7791dc85a0e513 not found: ID does not exist" containerID="95d75f0a15fd14dcffaa5f9df7cbaf02cfee031b1e824b81ab7791dc85a0e513"
Sep 16 12:41:22.903976 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.903873    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d75f0a15fd14dcffaa5f9df7cbaf02cfee031b1e824b81ab7791dc85a0e513"} err="failed to get container status \"95d75f0a15fd14dcffaa5f9df7cbaf02cfee031b1e824b81ab7791dc85a0e513\": rpc error: code = NotFound desc = could not find container \"95d75f0a15fd14dcffaa5f9df7cbaf02cfee031b1e824b81ab7791dc85a0e513\": container with ID starting with 95d75f0a15fd14dcffaa5f9df7cbaf02cfee031b1e824b81ab7791dc85a0e513 not found: ID does not exist"
Sep 16 12:41:22.903976 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.903893    2558 scope.go:117] "RemoveContainer" containerID="c2283f137bb2362e1afffc413eb9885659a28665d58335f93a5345f883b5efb6"
Sep 16 12:41:22.904188 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:41:22.904169    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2283f137bb2362e1afffc413eb9885659a28665d58335f93a5345f883b5efb6\": container with ID starting with c2283f137bb2362e1afffc413eb9885659a28665d58335f93a5345f883b5efb6 not found: ID does not exist" containerID="c2283f137bb2362e1afffc413eb9885659a28665d58335f93a5345f883b5efb6"
Sep 16 12:41:22.904230 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.904194    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2283f137bb2362e1afffc413eb9885659a28665d58335f93a5345f883b5efb6"} err="failed to get container status \"c2283f137bb2362e1afffc413eb9885659a28665d58335f93a5345f883b5efb6\": rpc error: code = NotFound desc = could not find container \"c2283f137bb2362e1afffc413eb9885659a28665d58335f93a5345f883b5efb6\": container with ID starting with c2283f137bb2362e1afffc413eb9885659a28665d58335f93a5345f883b5efb6 not found: ID does not exist"
Sep 16 12:41:22.905819 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.905799    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"]
Sep 16 12:41:22.909362 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:22.909336    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-89b7ca-openshift-default-795b96b54d-pz22t"]
Sep 16 12:41:23.758766 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:23.758728    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" path="/var/lib/kubelet/pods/af634e00-dcc3-4748-9e01-2fd6675f622f/volumes"
Sep 16 12:41:27.912397 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:27.912364    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c"]
Sep 16 12:41:27.914443 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:27.914419    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a9008-29826030-wch5c"]
Sep 16 12:41:27.925339 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:27.925308    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:28.005129 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:28.005103    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:28.109410 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:28.109383    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:28.209324 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:28.209241    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:28.512461 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:28.512377    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:28.614738 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:28.614701    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:29.346989 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:29.346960    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:29.758217 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:29.758136    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9548ccb9-050b-46db-8d90-0b0da20f932a" path="/var/lib/kubelet/pods/9548ccb9-050b-46db-8d90-0b0da20f932a/volumes"
Sep 16 12:41:34.712613 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:34.712554    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:34.819863 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:34.819825    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:34.971064 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:34.970983    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"]
Sep 16 12:41:34.971245 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:34.971218    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerName="istio-proxy" containerID="cri-o://dd4473f1b437683373353ee3cb2c1d64338ddc4a6e87c42a3dbf67d809f5f00b" gracePeriod=30
Sep 16 12:41:35.035617 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:35.035583    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:36.342283 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:36.342253    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:37.099228 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:37.099193    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:38.594468 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:38.594432    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:41:38.594868 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:38.594500    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:41:40.222341 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.222315    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:41:40.325105 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325005    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-workload-socket\") pod \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") "
Sep 16 12:41:40.325105 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325066    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9rzc\" (UniqueName: \"kubernetes.io/projected/6c8318df-1a44-49c3-bc52-ceb2d94a2320-kube-api-access-x9rzc\") pod \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") "
Sep 16 12:41:40.325105 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325100    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-data\") pod \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") "
Sep 16 12:41:40.325397 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325122    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istiod-ca-cert\") pod \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") "
Sep 16 12:41:40.325397 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325149    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-envoy\") pod \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") "
Sep 16 12:41:40.325397 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325212    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-token\") pod \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") "
Sep 16 12:41:40.325397 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325238    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-workload-certs\") pod \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") "
Sep 16 12:41:40.325397 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325286    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-podinfo\") pod \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") "
Sep 16 12:41:40.325397 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325312    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-credential-socket\") pod \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\" (UID: \"6c8318df-1a44-49c3-bc52-ceb2d94a2320\") "
Sep 16 12:41:40.325397 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325361    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c8318df-1a44-49c3-bc52-ceb2d94a2320" (UID: "6c8318df-1a44-49c3-bc52-ceb2d94a2320"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:40.325397 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325388    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c8318df-1a44-49c3-bc52-ceb2d94a2320" (UID: "6c8318df-1a44-49c3-bc52-ceb2d94a2320"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:40.325823 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325634    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-workload-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:40.325823 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325653    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-data\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:40.325823 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325666    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c8318df-1a44-49c3-bc52-ceb2d94a2320" (UID: "6c8318df-1a44-49c3-bc52-ceb2d94a2320"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:40.325823 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325681    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c8318df-1a44-49c3-bc52-ceb2d94a2320" (UID: "6c8318df-1a44-49c3-bc52-ceb2d94a2320"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:40.325823 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.325810    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c8318df-1a44-49c3-bc52-ceb2d94a2320" (UID: "6c8318df-1a44-49c3-bc52-ceb2d94a2320"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:40.327503 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.327468    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c8318df-1a44-49c3-bc52-ceb2d94a2320" (UID: "6c8318df-1a44-49c3-bc52-ceb2d94a2320"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:40.327629 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.327502    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8318df-1a44-49c3-bc52-ceb2d94a2320-kube-api-access-x9rzc" (OuterVolumeSpecName: "kube-api-access-x9rzc") pod "6c8318df-1a44-49c3-bc52-ceb2d94a2320" (UID: "6c8318df-1a44-49c3-bc52-ceb2d94a2320"). InnerVolumeSpecName "kube-api-access-x9rzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:40.327791 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.327772    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c8318df-1a44-49c3-bc52-ceb2d94a2320" (UID: "6c8318df-1a44-49c3-bc52-ceb2d94a2320"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:41:40.327836 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.327770    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c8318df-1a44-49c3-bc52-ceb2d94a2320" (UID: "6c8318df-1a44-49c3-bc52-ceb2d94a2320"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:40.427030 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.426986    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9rzc\" (UniqueName: \"kubernetes.io/projected/6c8318df-1a44-49c3-bc52-ceb2d94a2320-kube-api-access-x9rzc\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:40.427030 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.427020    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istiod-ca-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:40.427030 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.427036    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-envoy\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:40.427279 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.427046    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-token\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:40.427279 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.427059    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-workload-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:40.427279 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.427069    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8318df-1a44-49c3-bc52-ceb2d94a2320-istio-podinfo\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:40.427279 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.427080    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8318df-1a44-49c3-bc52-ceb2d94a2320-credential-socket\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:40.972307 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.972269    2558 generic.go:356] "Generic (PLEG): container finished" podID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerID="dd4473f1b437683373353ee3cb2c1d64338ddc4a6e87c42a3dbf67d809f5f00b" exitCode=0
Sep 16 12:41:40.972476 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.972330    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"
Sep 16 12:41:40.972476 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.972352    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" event={"ID":"6c8318df-1a44-49c3-bc52-ceb2d94a2320","Type":"ContainerDied","Data":"dd4473f1b437683373353ee3cb2c1d64338ddc4a6e87c42a3dbf67d809f5f00b"}
Sep 16 12:41:40.972476 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.972437    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km" event={"ID":"6c8318df-1a44-49c3-bc52-ceb2d94a2320","Type":"ContainerDied","Data":"197e7ed6ccf231f01c07d5b3dac951d5508f1867291895f134613c162fb753a1"}
Sep 16 12:41:40.972476 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.972460    2558 scope.go:117] "RemoveContainer" containerID="dd4473f1b437683373353ee3cb2c1d64338ddc4a6e87c42a3dbf67d809f5f00b"
Sep 16 12:41:40.981968 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.981948    2558 scope.go:117] "RemoveContainer" containerID="69806dd72ea2f753ca27fc451a416dfea49cd371e3a66fe9a08778177b5ca46a"
Sep 16 12:41:40.989860 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.989842    2558 scope.go:117] "RemoveContainer" containerID="dd4473f1b437683373353ee3cb2c1d64338ddc4a6e87c42a3dbf67d809f5f00b"
Sep 16 12:41:40.990139 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:41:40.990119    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4473f1b437683373353ee3cb2c1d64338ddc4a6e87c42a3dbf67d809f5f00b\": container with ID starting with dd4473f1b437683373353ee3cb2c1d64338ddc4a6e87c42a3dbf67d809f5f00b not found: ID does not exist" containerID="dd4473f1b437683373353ee3cb2c1d64338ddc4a6e87c42a3dbf67d809f5f00b"
Sep 16 12:41:40.990222 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.990148    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4473f1b437683373353ee3cb2c1d64338ddc4a6e87c42a3dbf67d809f5f00b"} err="failed to get container status \"dd4473f1b437683373353ee3cb2c1d64338ddc4a6e87c42a3dbf67d809f5f00b\": rpc error: code = NotFound desc = could not find container \"dd4473f1b437683373353ee3cb2c1d64338ddc4a6e87c42a3dbf67d809f5f00b\": container with ID starting with dd4473f1b437683373353ee3cb2c1d64338ddc4a6e87c42a3dbf67d809f5f00b not found: ID does not exist"
Sep 16 12:41:40.990222 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.990165    2558 scope.go:117] "RemoveContainer" containerID="69806dd72ea2f753ca27fc451a416dfea49cd371e3a66fe9a08778177b5ca46a"
Sep 16 12:41:40.990392 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:41:40.990376    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69806dd72ea2f753ca27fc451a416dfea49cd371e3a66fe9a08778177b5ca46a\": container with ID starting with 69806dd72ea2f753ca27fc451a416dfea49cd371e3a66fe9a08778177b5ca46a not found: ID does not exist" containerID="69806dd72ea2f753ca27fc451a416dfea49cd371e3a66fe9a08778177b5ca46a"
Sep 16 12:41:40.990433 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.990395    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69806dd72ea2f753ca27fc451a416dfea49cd371e3a66fe9a08778177b5ca46a"} err="failed to get container status \"69806dd72ea2f753ca27fc451a416dfea49cd371e3a66fe9a08778177b5ca46a\": rpc error: code = NotFound desc = could not find container \"69806dd72ea2f753ca27fc451a416dfea49cd371e3a66fe9a08778177b5ca46a\": container with ID starting with 69806dd72ea2f753ca27fc451a416dfea49cd371e3a66fe9a08778177b5ca46a not found: ID does not exist"
Sep 16 12:41:40.993798 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.993775    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"]
Sep 16 12:41:40.997742 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:40.997721    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a9008-openshift-default-55bf7997cb-7g7km"]
Sep 16 12:41:41.758043 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:41.758006    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" path="/var/lib/kubelet/pods/6c8318df-1a44-49c3-bc52-ceb2d94a2320/volumes"
Sep 16 12:41:46.026895 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:46.026858    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"]
Sep 16 12:41:46.027310 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:46.027077    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7" podUID="0b0a2c79-e014-4508-8d99-e3f51de91067" containerName="payload-processing" containerID="cri-o://9cedb4dcda1b1585bcec89d339ee5530a3b78afe468fbf1e5957f5bbb9e63abf" gracePeriod=30
Sep 16 12:41:46.186422 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:46.186370    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:46.283563 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:46.283485    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"
Sep 16 12:41:46.288475 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:46.288446    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:46.479818 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:46.479776    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcb5s\" (UniqueName: \"kubernetes.io/projected/0b0a2c79-e014-4508-8d99-e3f51de91067-kube-api-access-bcb5s\") pod \"0b0a2c79-e014-4508-8d99-e3f51de91067\" (UID: \"0b0a2c79-e014-4508-8d99-e3f51de91067\") "
Sep 16 12:41:46.479989 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:46.479834    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0a2c79-e014-4508-8d99-e3f51de91067-plugins-config-volume\") pod \"0b0a2c79-e014-4508-8d99-e3f51de91067\" (UID: \"0b0a2c79-e014-4508-8d99-e3f51de91067\") "
Sep 16 12:41:46.480221 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:46.480197    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0a2c79-e014-4508-8d99-e3f51de91067-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b0a2c79-e014-4508-8d99-e3f51de91067" (UID: "0b0a2c79-e014-4508-8d99-e3f51de91067"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:46.481955 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:46.481935    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0a2c79-e014-4508-8d99-e3f51de91067-kube-api-access-bcb5s" (OuterVolumeSpecName: "kube-api-access-bcb5s") pod "0b0a2c79-e014-4508-8d99-e3f51de91067" (UID: "0b0a2c79-e014-4508-8d99-e3f51de91067"). InnerVolumeSpecName "kube-api-access-bcb5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:46.561825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:46.561719    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:46.581375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:46.581337    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcb5s\" (UniqueName: \"kubernetes.io/projected/0b0a2c79-e014-4508-8d99-e3f51de91067-kube-api-access-bcb5s\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:46.581375 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:46.581368    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0a2c79-e014-4508-8d99-e3f51de91067-plugins-config-volume\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:46.661451 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:46.661398    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:47.009808 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:47.009768    2558 generic.go:356] "Generic (PLEG): container finished" podID="0b0a2c79-e014-4508-8d99-e3f51de91067" containerID="9cedb4dcda1b1585bcec89d339ee5530a3b78afe468fbf1e5957f5bbb9e63abf" exitCode=0
Sep 16 12:41:47.009995 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:47.009819    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7" event={"ID":"0b0a2c79-e014-4508-8d99-e3f51de91067","Type":"ContainerDied","Data":"9cedb4dcda1b1585bcec89d339ee5530a3b78afe468fbf1e5957f5bbb9e63abf"}
Sep 16 12:41:47.009995 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:47.009830    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"
Sep 16 12:41:47.009995 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:47.009900    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7" event={"ID":"0b0a2c79-e014-4508-8d99-e3f51de91067","Type":"ContainerDied","Data":"dbd65ae9b7838ef4b4d0346a233a5f607a6a648e09dcd42523a2c1938c2032fe"}
Sep 16 12:41:47.009995 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:47.009932    2558 scope.go:117] "RemoveContainer" containerID="9cedb4dcda1b1585bcec89d339ee5530a3b78afe468fbf1e5957f5bbb9e63abf"
Sep 16 12:41:47.019239 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:47.019218    2558 scope.go:117] "RemoveContainer" containerID="9cedb4dcda1b1585bcec89d339ee5530a3b78afe468fbf1e5957f5bbb9e63abf"
Sep 16 12:41:47.019532 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:41:47.019502    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cedb4dcda1b1585bcec89d339ee5530a3b78afe468fbf1e5957f5bbb9e63abf\": container with ID starting with 9cedb4dcda1b1585bcec89d339ee5530a3b78afe468fbf1e5957f5bbb9e63abf not found: ID does not exist" containerID="9cedb4dcda1b1585bcec89d339ee5530a3b78afe468fbf1e5957f5bbb9e63abf"
Sep 16 12:41:47.019631 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:47.019533    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cedb4dcda1b1585bcec89d339ee5530a3b78afe468fbf1e5957f5bbb9e63abf"} err="failed to get container status \"9cedb4dcda1b1585bcec89d339ee5530a3b78afe468fbf1e5957f5bbb9e63abf\": rpc error: code = NotFound desc = could not find container \"9cedb4dcda1b1585bcec89d339ee5530a3b78afe468fbf1e5957f5bbb9e63abf\": container with ID starting with 9cedb4dcda1b1585bcec89d339ee5530a3b78afe468fbf1e5957f5bbb9e63abf not found: ID does not exist"
Sep 16 12:41:47.030731 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:47.030705    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"]
Sep 16 12:41:47.034839 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:47.034815    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e94e5-7785856677-fdzh7"]
Sep 16 12:41:47.757903 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:47.757873    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b0a2c79-e014-4508-8d99-e3f51de91067" path="/var/lib/kubelet/pods/0b0a2c79-e014-4508-8d99-e3f51de91067/volumes"
Sep 16 12:41:52.766738 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:52.766704    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:52.831423 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:52.831387    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:53.028195 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:53.028115    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:57.907157 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:57.907117    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-774dfb4f47-h2hrq"]
Sep 16 12:41:57.907585 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:57.907346    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-774dfb4f47-h2hrq" podUID="f7a8d41b-9a6b-42cb-832e-af64d1604ddd" containerName="manager" containerID="cri-o://a6a635d0c1ee6e1d1cc9aff56c7d165483e07e71732f4a36ddd3d2a5ece732e8" gracePeriod=30
Sep 16 12:41:58.065693 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:58.065659    2558 generic.go:356] "Generic (PLEG): container finished" podID="f7a8d41b-9a6b-42cb-832e-af64d1604ddd" containerID="a6a635d0c1ee6e1d1cc9aff56c7d165483e07e71732f4a36ddd3d2a5ece732e8" exitCode=0
Sep 16 12:41:58.065871 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:58.065740    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774dfb4f47-h2hrq" event={"ID":"f7a8d41b-9a6b-42cb-832e-af64d1604ddd","Type":"ContainerDied","Data":"a6a635d0c1ee6e1d1cc9aff56c7d165483e07e71732f4a36ddd3d2a5ece732e8"}
Sep 16 12:41:58.159576 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:58.159485    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:41:58.176673 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:58.176640    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-metrics-certs\") pod \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\" (UID: \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\") "
Sep 16 12:41:58.176864 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:58.176754    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bnjj\" (UniqueName: \"kubernetes.io/projected/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-kube-api-access-6bnjj\") pod \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\" (UID: \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\") "
Sep 16 12:41:58.176864 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:58.176812    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-webhook-cert\") pod \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\" (UID: \"f7a8d41b-9a6b-42cb-832e-af64d1604ddd\") "
Sep 16 12:41:58.178770 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:58.178739    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f7a8d41b-9a6b-42cb-832e-af64d1604ddd" (UID: "f7a8d41b-9a6b-42cb-832e-af64d1604ddd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:58.178770 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:58.178761    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f7a8d41b-9a6b-42cb-832e-af64d1604ddd" (UID: "f7a8d41b-9a6b-42cb-832e-af64d1604ddd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:58.178915 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:58.178879    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-kube-api-access-6bnjj" (OuterVolumeSpecName: "kube-api-access-6bnjj") pod "f7a8d41b-9a6b-42cb-832e-af64d1604ddd" (UID: "f7a8d41b-9a6b-42cb-832e-af64d1604ddd"). InnerVolumeSpecName "kube-api-access-6bnjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:58.278036 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:58.277995    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-metrics-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:58.278199 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:58.278041    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bnjj\" (UniqueName: \"kubernetes.io/projected/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-kube-api-access-6bnjj\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:58.278199 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:58.278060    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7a8d41b-9a6b-42cb-832e-af64d1604ddd-webhook-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:41:59.072644 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:59.072595    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774dfb4f47-h2hrq" event={"ID":"f7a8d41b-9a6b-42cb-832e-af64d1604ddd","Type":"ContainerDied","Data":"232d8975312fb741a0f08019f7410e53b5b9c86c39b0f02ce2d9535a7c1620d3"}
Sep 16 12:41:59.072644 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:59.072644    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774dfb4f47-h2hrq"
Sep 16 12:41:59.073186 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:59.072708    2558 scope.go:117] "RemoveContainer" containerID="a6a635d0c1ee6e1d1cc9aff56c7d165483e07e71732f4a36ddd3d2a5ece732e8"
Sep 16 12:41:59.094868 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:59.094836    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-774dfb4f47-h2hrq"]
Sep 16 12:41:59.098656 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:59.098627    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-774dfb4f47-h2hrq"]
Sep 16 12:41:59.757158 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:59.757118    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7a8d41b-9a6b-42cb-832e-af64d1604ddd" path="/var/lib/kubelet/pods/f7a8d41b-9a6b-42cb-832e-af64d1604ddd/volumes"
Sep 16 12:41:59.910862 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:41:59.910830    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:42:00.012231 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:00.012146    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:42:24.057303 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:24.057226    2558 scope.go:117] "RemoveContainer" containerID="cca1792f4c667a65ba919f1a788152cff1c694ace42b567c8ae750424ad8f0d6"
Sep 16 12:42:59.877181 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877136    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-774dfb4f47-m9zls"]
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877669    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b33817-756f-44a7-8f63-6e1d2d07be86" containerName="payload-pre-processing"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877690    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b33817-756f-44a7-8f63-6e1d2d07be86" containerName="payload-pre-processing"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877707    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerName="istio-proxy"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877715    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerName="istio-proxy"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877725    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerName="istio-proxy"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877735    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerName="istio-proxy"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877745    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7a8d41b-9a6b-42cb-832e-af64d1604ddd" containerName="manager"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877753    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7a8d41b-9a6b-42cb-832e-af64d1604ddd" containerName="manager"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877765    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerName="istio-proxy"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877774    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerName="istio-proxy"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877789    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerName="istio-proxy"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877797    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerName="istio-proxy"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877805    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerName="istio-proxy"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877813    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerName="istio-proxy"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877825    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0a2c79-e014-4508-8d99-e3f51de91067" containerName="payload-processing"
Sep 16 12:42:59.877825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877834    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0a2c79-e014-4508-8d99-e3f51de91067" containerName="payload-processing"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877843    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerName="istio-proxy"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877850    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerName="istio-proxy"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877862    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerName="istio-proxy"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877873    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerName="istio-proxy"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877886    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf7908c-8e7c-4c5a-802b-a6bceb64075a" containerName="payload-processing"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877894    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf7908c-8e7c-4c5a-802b-a6bceb64075a" containerName="payload-processing"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877915    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a1da320-b39d-428f-a1ea-385d9bd1577a" containerName="maas-api"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.877923    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a1da320-b39d-428f-a1ea-385d9bd1577a" containerName="maas-api"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.878011    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerName="istio-proxy"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.878023    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerName="istio-proxy"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.878032    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf7908c-8e7c-4c5a-802b-a6bceb64075a" containerName="payload-processing"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.878041    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="22b33817-756f-44a7-8f63-6e1d2d07be86" containerName="payload-pre-processing"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.878050    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerName="istio-proxy"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.878058    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7a8d41b-9a6b-42cb-832e-af64d1604ddd" containerName="manager"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.878065    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerName="istio-proxy"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.878074    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffade955-1f51-4d20-9d4f-cedb751cf42b" containerName="istio-proxy"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.878083    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c8318df-1a44-49c3-bc52-ceb2d94a2320" containerName="istio-proxy"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.878091    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a1da320-b39d-428f-a1ea-385d9bd1577a" containerName="maas-api"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.878098    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a01c434a-188c-4cc4-a835-1fbf833440d5" containerName="istio-proxy"
Sep 16 12:42:59.878633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.878106    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0a2c79-e014-4508-8d99-e3f51de91067" containerName="payload-processing"
Sep 16 12:42:59.881418 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.881397    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:42:59.884766 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.884740    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 12:42:59.884900 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.884744    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 12:42:59.884900 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.884821    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mpg4c\""
Sep 16 12:42:59.884900 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.884843    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 12:42:59.891643 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.891615    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-774dfb4f47-m9zls"]
Sep 16 12:42:59.917430 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.917396    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4lk\" (UniqueName: \"kubernetes.io/projected/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-kube-api-access-sg4lk\") pod \"maas-controller-774dfb4f47-m9zls\" (UID: \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\") " pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:42:59.917645 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.917469    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-webhook-cert\") pod \"maas-controller-774dfb4f47-m9zls\" (UID: \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\") " pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:42:59.917645 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:42:59.917496    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-metrics-certs\") pod \"maas-controller-774dfb4f47-m9zls\" (UID: \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\") " pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:43:00.018513 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:00.018471    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4lk\" (UniqueName: \"kubernetes.io/projected/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-kube-api-access-sg4lk\") pod \"maas-controller-774dfb4f47-m9zls\" (UID: \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\") " pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:43:00.018726 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:00.018533    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-webhook-cert\") pod \"maas-controller-774dfb4f47-m9zls\" (UID: \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\") " pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:43:00.018726 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:00.018577    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-metrics-certs\") pod \"maas-controller-774dfb4f47-m9zls\" (UID: \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\") " pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:43:00.021071 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:00.021043    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-webhook-cert\") pod \"maas-controller-774dfb4f47-m9zls\" (UID: \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\") " pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:43:00.021191 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:00.021145    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-metrics-certs\") pod \"maas-controller-774dfb4f47-m9zls\" (UID: \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\") " pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:43:00.026367 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:00.026346    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4lk\" (UniqueName: \"kubernetes.io/projected/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-kube-api-access-sg4lk\") pod \"maas-controller-774dfb4f47-m9zls\" (UID: \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\") " pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:43:00.193426 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:00.193309    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:43:00.467738 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:00.467708    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-774dfb4f47-m9zls"]
Sep 16 12:43:00.470126 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:43:00.470099    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7f3e28_0b5a_4b8a_b4c4_3c9685bb2473.slice/crio-bc7d1ed277c91b420e84e237e6f2e967020d21cfba8c40ecbfbd260304dc90d2 WatchSource:0}: Error finding container bc7d1ed277c91b420e84e237e6f2e967020d21cfba8c40ecbfbd260304dc90d2: Status 404 returned error can't find the container with id bc7d1ed277c91b420e84e237e6f2e967020d21cfba8c40ecbfbd260304dc90d2
Sep 16 12:43:00.471388 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:00.471370    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:43:01.357086 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:01.357050    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774dfb4f47-m9zls" event={"ID":"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473","Type":"ContainerStarted","Data":"bc7d1ed277c91b420e84e237e6f2e967020d21cfba8c40ecbfbd260304dc90d2"}
Sep 16 12:43:04.373057 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:04.373015    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774dfb4f47-m9zls" event={"ID":"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473","Type":"ContainerStarted","Data":"a48a37c975351a395a792f99eaffc551b488601b0c96f177e2ef43c1b2aa86fd"}
Sep 16 12:43:04.373440 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:04.373085    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:43:04.393586 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:04.393520    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-774dfb4f47-m9zls" podStartSLOduration=2.53857488 podStartE2EDuration="5.393502276s" totalImagesPullingTime="2.854927396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:43:04.392159076 +0000 UTC m=+2381.146117801" watchObservedRunningTime="2026-09-16 12:43:04.393502276 +0000 UTC m=+2381.147461007"
Sep 16 12:43:15.380244 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:15.380211    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:43:21.246075 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:21.246039    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:43:21.305107 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:21.305075    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:43:22.433237 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:22.433198    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:43:22.523824 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:22.523783    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-774dfb4f47-m9zls"]
Sep 16 12:43:22.524078 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:22.524051    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-774dfb4f47-m9zls" podUID="0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473" containerName="manager" containerID="cri-o://a48a37c975351a395a792f99eaffc551b488601b0c96f177e2ef43c1b2aa86fd" gracePeriod=30
Sep 16 12:43:22.773442 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:22.773409    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:43:22.920123 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:22.920089    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-webhook-cert\") pod \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\" (UID: \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\") "
Sep 16 12:43:22.920338 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:22.920134    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-metrics-certs\") pod \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\" (UID: \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\") "
Sep 16 12:43:22.920338 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:22.920157    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg4lk\" (UniqueName: \"kubernetes.io/projected/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-kube-api-access-sg4lk\") pod \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\" (UID: \"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473\") "
Sep 16 12:43:22.922337 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:22.922302    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-kube-api-access-sg4lk" (OuterVolumeSpecName: "kube-api-access-sg4lk") pod "0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473" (UID: "0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473"). InnerVolumeSpecName "kube-api-access-sg4lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:43:22.922450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:22.922340    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473" (UID: "0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:43:22.922450 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:22.922352    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473" (UID: "0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:43:23.021893 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:23.021796    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-webhook-cert\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:43:23.021893 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:23.021823    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-metrics-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:43:23.021893 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:23.021836    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg4lk\" (UniqueName: \"kubernetes.io/projected/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473-kube-api-access-sg4lk\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:43:23.467316 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:23.467273    2558 generic.go:356] "Generic (PLEG): container finished" podID="0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473" containerID="a48a37c975351a395a792f99eaffc551b488601b0c96f177e2ef43c1b2aa86fd" exitCode=0
Sep 16 12:43:23.467316 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:23.467320    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774dfb4f47-m9zls"
Sep 16 12:43:23.467905 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:23.467360    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774dfb4f47-m9zls" event={"ID":"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473","Type":"ContainerDied","Data":"a48a37c975351a395a792f99eaffc551b488601b0c96f177e2ef43c1b2aa86fd"}
Sep 16 12:43:23.467905 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:23.467426    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774dfb4f47-m9zls" event={"ID":"0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473","Type":"ContainerDied","Data":"bc7d1ed277c91b420e84e237e6f2e967020d21cfba8c40ecbfbd260304dc90d2"}
Sep 16 12:43:23.467905 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:23.467459    2558 scope.go:117] "RemoveContainer" containerID="a48a37c975351a395a792f99eaffc551b488601b0c96f177e2ef43c1b2aa86fd"
Sep 16 12:43:23.476604 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:23.476583    2558 scope.go:117] "RemoveContainer" containerID="a48a37c975351a395a792f99eaffc551b488601b0c96f177e2ef43c1b2aa86fd"
Sep 16 12:43:23.476868 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:43:23.476847    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48a37c975351a395a792f99eaffc551b488601b0c96f177e2ef43c1b2aa86fd\": container with ID starting with a48a37c975351a395a792f99eaffc551b488601b0c96f177e2ef43c1b2aa86fd not found: ID does not exist" containerID="a48a37c975351a395a792f99eaffc551b488601b0c96f177e2ef43c1b2aa86fd"
Sep 16 12:43:23.476920 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:23.476880    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48a37c975351a395a792f99eaffc551b488601b0c96f177e2ef43c1b2aa86fd"} err="failed to get container status \"a48a37c975351a395a792f99eaffc551b488601b0c96f177e2ef43c1b2aa86fd\": rpc error: code = NotFound desc = could not find container \"a48a37c975351a395a792f99eaffc551b488601b0c96f177e2ef43c1b2aa86fd\": container with ID starting with a48a37c975351a395a792f99eaffc551b488601b0c96f177e2ef43c1b2aa86fd not found: ID does not exist"
Sep 16 12:43:23.489604 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:23.489579    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-774dfb4f47-m9zls"]
Sep 16 12:43:23.493438 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:23.493414    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-774dfb4f47-m9zls"]
Sep 16 12:43:23.757978 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:23.757895    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473" path="/var/lib/kubelet/pods/0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473/volumes"
Sep 16 12:43:24.269901 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.269862    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:43:24.372258 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.372173    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-774dfb4f47-jg8h6"]
Sep 16 12:43:24.372638 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.372620    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerName="istio-proxy"
Sep 16 12:43:24.372749 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.372640    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerName="istio-proxy"
Sep 16 12:43:24.372749 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.372651    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473" containerName="manager"
Sep 16 12:43:24.372749 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.372659    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473" containerName="manager"
Sep 16 12:43:24.372906 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.372774    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d7f3e28-0b5a-4b8a-b4c4-3c9685bb2473" containerName="manager"
Sep 16 12:43:24.372906 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.372788    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="af634e00-dcc3-4748-9e01-2fd6675f622f" containerName="istio-proxy"
Sep 16 12:43:24.376982 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.376959    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774dfb4f47-jg8h6"
Sep 16 12:43:24.380051 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.380024    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mpg4c\""
Sep 16 12:43:24.380051 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.380047    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 12:43:24.380231 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.380063    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 12:43:24.380231 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.380063    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 12:43:24.385216 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.385191    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-774dfb4f47-jg8h6"]
Sep 16 12:43:24.535672 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.535618    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e974f342-8f7d-46fd-a589-e9f2f7da4c44-metrics-certs\") pod \"maas-controller-774dfb4f47-jg8h6\" (UID: \"e974f342-8f7d-46fd-a589-e9f2f7da4c44\") " pod="opendatahub/maas-controller-774dfb4f47-jg8h6"
Sep 16 12:43:24.536099 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.535690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd9qm\" (UniqueName: \"kubernetes.io/projected/e974f342-8f7d-46fd-a589-e9f2f7da4c44-kube-api-access-dd9qm\") pod \"maas-controller-774dfb4f47-jg8h6\" (UID: \"e974f342-8f7d-46fd-a589-e9f2f7da4c44\") " pod="opendatahub/maas-controller-774dfb4f47-jg8h6"
Sep 16 12:43:24.536099 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.535747    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e974f342-8f7d-46fd-a589-e9f2f7da4c44-webhook-cert\") pod \"maas-controller-774dfb4f47-jg8h6\" (UID: \"e974f342-8f7d-46fd-a589-e9f2f7da4c44\") " pod="opendatahub/maas-controller-774dfb4f47-jg8h6"
Sep 16 12:43:24.636979 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.636874    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e974f342-8f7d-46fd-a589-e9f2f7da4c44-metrics-certs\") pod \"maas-controller-774dfb4f47-jg8h6\" (UID: \"e974f342-8f7d-46fd-a589-e9f2f7da4c44\") " pod="opendatahub/maas-controller-774dfb4f47-jg8h6"
Sep 16 12:43:24.636979 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.636938    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd9qm\" (UniqueName: \"kubernetes.io/projected/e974f342-8f7d-46fd-a589-e9f2f7da4c44-kube-api-access-dd9qm\") pod \"maas-controller-774dfb4f47-jg8h6\" (UID: \"e974f342-8f7d-46fd-a589-e9f2f7da4c44\") " pod="opendatahub/maas-controller-774dfb4f47-jg8h6"
Sep 16 12:43:24.637221 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.637001    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e974f342-8f7d-46fd-a589-e9f2f7da4c44-webhook-cert\") pod \"maas-controller-774dfb4f47-jg8h6\" (UID: \"e974f342-8f7d-46fd-a589-e9f2f7da4c44\") " pod="opendatahub/maas-controller-774dfb4f47-jg8h6"
Sep 16 12:43:24.639468 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.639446    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e974f342-8f7d-46fd-a589-e9f2f7da4c44-webhook-cert\") pod \"maas-controller-774dfb4f47-jg8h6\" (UID: \"e974f342-8f7d-46fd-a589-e9f2f7da4c44\") " pod="opendatahub/maas-controller-774dfb4f47-jg8h6"
Sep 16 12:43:24.639608 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.639586    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e974f342-8f7d-46fd-a589-e9f2f7da4c44-metrics-certs\") pod \"maas-controller-774dfb4f47-jg8h6\" (UID: \"e974f342-8f7d-46fd-a589-e9f2f7da4c44\") " pod="opendatahub/maas-controller-774dfb4f47-jg8h6"
Sep 16 12:43:24.649172 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.649149    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd9qm\" (UniqueName: \"kubernetes.io/projected/e974f342-8f7d-46fd-a589-e9f2f7da4c44-kube-api-access-dd9qm\") pod \"maas-controller-774dfb4f47-jg8h6\" (UID: \"e974f342-8f7d-46fd-a589-e9f2f7da4c44\") " pod="opendatahub/maas-controller-774dfb4f47-jg8h6"
Sep 16 12:43:24.688169 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.688129    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774dfb4f47-jg8h6"
Sep 16 12:43:24.957808 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:24.957777    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-774dfb4f47-jg8h6"]
Sep 16 12:43:24.960031 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:43:24.960008    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode974f342_8f7d_46fd_a589_e9f2f7da4c44.slice/crio-91487be3f483c0128bb156944e1601a4eacb07edd15fa38cb4053bc687279857 WatchSource:0}: Error finding container 91487be3f483c0128bb156944e1601a4eacb07edd15fa38cb4053bc687279857: Status 404 returned error can't find the container with id 91487be3f483c0128bb156944e1601a4eacb07edd15fa38cb4053bc687279857
Sep 16 12:43:25.477420 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:25.477388    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774dfb4f47-jg8h6" event={"ID":"e974f342-8f7d-46fd-a589-e9f2f7da4c44","Type":"ContainerStarted","Data":"91487be3f483c0128bb156944e1601a4eacb07edd15fa38cb4053bc687279857"}
Sep 16 12:43:26.483909 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:26.483876    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774dfb4f47-jg8h6" event={"ID":"e974f342-8f7d-46fd-a589-e9f2f7da4c44","Type":"ContainerStarted","Data":"971c9967361bd10e65e9355b8569fe27455f8f26865dee87b45448a2ae1dfcad"}
Sep 16 12:43:26.484410 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:26.483972    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-774dfb4f47-jg8h6"
Sep 16 12:43:26.502644 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:26.502586    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-774dfb4f47-jg8h6" podStartSLOduration=1.853104676 podStartE2EDuration="2.502567474s" totalImagesPullingTime="649.462798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:43:26.500454398 +0000 UTC m=+2403.254413129" watchObservedRunningTime="2026-09-16 12:43:26.502567474 +0000 UTC m=+2403.256526203"
Sep 16 12:43:34.208333 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:34.208298    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:43:37.491592 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:37.491540    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-774dfb4f47-jg8h6"
Sep 16 12:43:45.599406 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:43:45.599367    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:44:02.830582 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:44:02.830529    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:45:56.391757 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:56.391719    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr"]
Sep 16 12:45:56.395124 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:56.395105    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr"
Sep 16 12:45:56.399115 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:56.399087    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mbkvc\""
Sep 16 12:45:56.412314 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:56.412284    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr"]
Sep 16 12:45:56.477380 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:56.477343    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/891012d0-fe51-436f-89e0-2b3983d51438-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fx6rr\" (UID: \"891012d0-fe51-436f-89e0-2b3983d51438\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr"
Sep 16 12:45:56.477626 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:56.477474    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlkzf\" (UniqueName: \"kubernetes.io/projected/891012d0-fe51-436f-89e0-2b3983d51438-kube-api-access-tlkzf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fx6rr\" (UID: \"891012d0-fe51-436f-89e0-2b3983d51438\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr"
Sep 16 12:45:56.578248 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:56.578190    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlkzf\" (UniqueName: \"kubernetes.io/projected/891012d0-fe51-436f-89e0-2b3983d51438-kube-api-access-tlkzf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fx6rr\" (UID: \"891012d0-fe51-436f-89e0-2b3983d51438\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr"
Sep 16 12:45:56.578443 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:56.578267    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/891012d0-fe51-436f-89e0-2b3983d51438-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fx6rr\" (UID: \"891012d0-fe51-436f-89e0-2b3983d51438\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr"
Sep 16 12:45:56.578725 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:56.578704    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/891012d0-fe51-436f-89e0-2b3983d51438-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fx6rr\" (UID: \"891012d0-fe51-436f-89e0-2b3983d51438\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr"
Sep 16 12:45:56.591063 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:56.591035    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlkzf\" (UniqueName: \"kubernetes.io/projected/891012d0-fe51-436f-89e0-2b3983d51438-kube-api-access-tlkzf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fx6rr\" (UID: \"891012d0-fe51-436f-89e0-2b3983d51438\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr"
Sep 16 12:45:56.705413 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:56.705319    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr"
Sep 16 12:45:56.985975 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:56.985945    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr"]
Sep 16 12:45:56.988422 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:45:56.988388    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891012d0_fe51_436f_89e0_2b3983d51438.slice/crio-d1a1a1cd235550025104ffeb28010607decab2e7e1fe9cbe95595252a5547c8b WatchSource:0}: Error finding container d1a1a1cd235550025104ffeb28010607decab2e7e1fe9cbe95595252a5547c8b: Status 404 returned error can't find the container with id d1a1a1cd235550025104ffeb28010607decab2e7e1fe9cbe95595252a5547c8b
Sep 16 12:45:57.183142 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:57.183104    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr" event={"ID":"891012d0-fe51-436f-89e0-2b3983d51438","Type":"ContainerStarted","Data":"0f28f6508d28a7e6691fb4b8140fa64af4d1322e2f44e5bb8f5f22c9fbf8d0fb"}
Sep 16 12:45:57.183294 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:57.183168    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr"
Sep 16 12:45:57.183294 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:57.183187    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr" event={"ID":"891012d0-fe51-436f-89e0-2b3983d51438","Type":"ContainerStarted","Data":"d1a1a1cd235550025104ffeb28010607decab2e7e1fe9cbe95595252a5547c8b"}
Sep 16 12:45:57.212071 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:45:57.212017    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr" podStartSLOduration=1.212000049 podStartE2EDuration="1.212000049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:45:57.209857174 +0000 UTC m=+2553.963815904" watchObservedRunningTime="2026-09-16 12:45:57.212000049 +0000 UTC m=+2553.965958779"
Sep 16 12:46:08.190452 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:46:08.190416    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fx6rr"
Sep 16 12:46:38.333365 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:46:38.333329    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:46:46.880646 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:46:46.880565    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:47:27.302966 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:27.302905    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:47:29.166755 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:29.166721    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:47:43.530405 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:43.530371    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:47:55.777790 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.777756    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"]
Sep 16 12:47:55.780882 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.780857    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:55.792035 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.792008    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"]
Sep 16 12:47:55.826871 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.826832    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f389a0e9-bea4-4a4f-ba15-138ababd43b4-maas-api-tls\") pod \"maas-api-5bfbbdfcb6-hgt87\" (UID: \"f389a0e9-bea4-4a4f-ba15-138ababd43b4\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:55.827044 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.826890    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f389a0e9-bea4-4a4f-ba15-138ababd43b4-openshift-service-ca\") pod \"maas-api-5bfbbdfcb6-hgt87\" (UID: \"f389a0e9-bea4-4a4f-ba15-138ababd43b4\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:55.827044 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.826961    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f389a0e9-bea4-4a4f-ba15-138ababd43b4-metrics-certs\") pod \"maas-api-5bfbbdfcb6-hgt87\" (UID: \"f389a0e9-bea4-4a4f-ba15-138ababd43b4\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:55.827044 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.826981    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vbpv\" (UniqueName: \"kubernetes.io/projected/f389a0e9-bea4-4a4f-ba15-138ababd43b4-kube-api-access-7vbpv\") pod \"maas-api-5bfbbdfcb6-hgt87\" (UID: \"f389a0e9-bea4-4a4f-ba15-138ababd43b4\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:55.927899 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.927856    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f389a0e9-bea4-4a4f-ba15-138ababd43b4-openshift-service-ca\") pod \"maas-api-5bfbbdfcb6-hgt87\" (UID: \"f389a0e9-bea4-4a4f-ba15-138ababd43b4\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:55.928089 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.927911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f389a0e9-bea4-4a4f-ba15-138ababd43b4-metrics-certs\") pod \"maas-api-5bfbbdfcb6-hgt87\" (UID: \"f389a0e9-bea4-4a4f-ba15-138ababd43b4\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:55.928089 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.927931    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vbpv\" (UniqueName: \"kubernetes.io/projected/f389a0e9-bea4-4a4f-ba15-138ababd43b4-kube-api-access-7vbpv\") pod \"maas-api-5bfbbdfcb6-hgt87\" (UID: \"f389a0e9-bea4-4a4f-ba15-138ababd43b4\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:55.928089 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.927983    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f389a0e9-bea4-4a4f-ba15-138ababd43b4-maas-api-tls\") pod \"maas-api-5bfbbdfcb6-hgt87\" (UID: \"f389a0e9-bea4-4a4f-ba15-138ababd43b4\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:55.928683 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.928655    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f389a0e9-bea4-4a4f-ba15-138ababd43b4-openshift-service-ca\") pod \"maas-api-5bfbbdfcb6-hgt87\" (UID: \"f389a0e9-bea4-4a4f-ba15-138ababd43b4\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:55.930404 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.930378    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f389a0e9-bea4-4a4f-ba15-138ababd43b4-maas-api-tls\") pod \"maas-api-5bfbbdfcb6-hgt87\" (UID: \"f389a0e9-bea4-4a4f-ba15-138ababd43b4\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:55.930485 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.930471    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f389a0e9-bea4-4a4f-ba15-138ababd43b4-metrics-certs\") pod \"maas-api-5bfbbdfcb6-hgt87\" (UID: \"f389a0e9-bea4-4a4f-ba15-138ababd43b4\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:55.935792 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:55.935770    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vbpv\" (UniqueName: \"kubernetes.io/projected/f389a0e9-bea4-4a4f-ba15-138ababd43b4-kube-api-access-7vbpv\") pod \"maas-api-5bfbbdfcb6-hgt87\" (UID: \"f389a0e9-bea4-4a4f-ba15-138ababd43b4\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:56.091381 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:56.091339    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:56.368460 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:56.368434    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"]
Sep 16 12:47:56.370395 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:47:56.370369    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf389a0e9_bea4_4a4f_ba15_138ababd43b4.slice/crio-123879532bafac9838ba77ccb2d0317d17b14d4b2b8dbce6f8ba8d77440e2856 WatchSource:0}: Error finding container 123879532bafac9838ba77ccb2d0317d17b14d4b2b8dbce6f8ba8d77440e2856: Status 404 returned error can't find the container with id 123879532bafac9838ba77ccb2d0317d17b14d4b2b8dbce6f8ba8d77440e2856
Sep 16 12:47:56.745288 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:56.745254    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87" event={"ID":"f389a0e9-bea4-4a4f-ba15-138ababd43b4","Type":"ContainerStarted","Data":"123879532bafac9838ba77ccb2d0317d17b14d4b2b8dbce6f8ba8d77440e2856"}
Sep 16 12:47:57.758077 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:57.758039    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87" event={"ID":"f389a0e9-bea4-4a4f-ba15-138ababd43b4","Type":"ContainerStarted","Data":"f9fef7a13d901d246f436497abc58d474ab97f5b97079eefa042436c30053a87"}
Sep 16 12:47:57.758460 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:57.758107    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:47:57.771999 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:47:57.771951    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87" podStartSLOduration=2.425650027 podStartE2EDuration="2.771934976s" totalImagesPullingTime="346.284949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:47:57.7710075 +0000 UTC m=+2674.524966232" watchObservedRunningTime="2026-09-16 12:47:57.771934976 +0000 UTC m=+2674.525893706"
Sep 16 12:48:03.765257 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:03.765226    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-hgt87"
Sep 16 12:48:03.813633 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:03.813600    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"]
Sep 16 12:48:03.813893 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:03.813869    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv" podUID="94a11a9d-b74b-4ae6-b35f-c6bea61d5403" containerName="maas-api" containerID="cri-o://442f784ae354c3164a39c20dee3a7173b874085620725da8f60a70d68de386dc" gracePeriod=30
Sep 16 12:48:05.062350 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.062322    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:48:05.120087 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.120054    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-maas-api-tls\") pod \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") "
Sep 16 12:48:05.120087 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.120098    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-metrics-certs\") pod \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") "
Sep 16 12:48:05.120347 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.120144    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhw9t\" (UniqueName: \"kubernetes.io/projected/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-kube-api-access-hhw9t\") pod \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") "
Sep 16 12:48:05.120347 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.120205    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-openshift-service-ca\") pod \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\" (UID: \"94a11a9d-b74b-4ae6-b35f-c6bea61d5403\") "
Sep 16 12:48:05.120651 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.120623    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94a11a9d-b74b-4ae6-b35f-c6bea61d5403" (UID: "94a11a9d-b74b-4ae6-b35f-c6bea61d5403"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:48:05.122262 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.122226    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-kube-api-access-hhw9t" (OuterVolumeSpecName: "kube-api-access-hhw9t") pod "94a11a9d-b74b-4ae6-b35f-c6bea61d5403" (UID: "94a11a9d-b74b-4ae6-b35f-c6bea61d5403"). InnerVolumeSpecName "kube-api-access-hhw9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:48:05.122262 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.122251    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "94a11a9d-b74b-4ae6-b35f-c6bea61d5403" (UID: "94a11a9d-b74b-4ae6-b35f-c6bea61d5403"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:48:05.122398 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.122274    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94a11a9d-b74b-4ae6-b35f-c6bea61d5403" (UID: "94a11a9d-b74b-4ae6-b35f-c6bea61d5403"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:48:05.221909 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.221813    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-maas-api-tls\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:48:05.221909 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.221848    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-metrics-certs\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:48:05.221909 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.221861    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhw9t\" (UniqueName: \"kubernetes.io/projected/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-kube-api-access-hhw9t\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:48:05.221909 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.221874    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a11a9d-b74b-4ae6-b35f-c6bea61d5403-openshift-service-ca\") on node \"ip-10-0-136-98.ec2.internal\" DevicePath \"\""
Sep 16 12:48:05.791899 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.791862    2558 generic.go:356] "Generic (PLEG): container finished" podID="94a11a9d-b74b-4ae6-b35f-c6bea61d5403" containerID="442f784ae354c3164a39c20dee3a7173b874085620725da8f60a70d68de386dc" exitCode=0
Sep 16 12:48:05.792052 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.791904    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"
Sep 16 12:48:05.792052 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.791933    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv" event={"ID":"94a11a9d-b74b-4ae6-b35f-c6bea61d5403","Type":"ContainerDied","Data":"442f784ae354c3164a39c20dee3a7173b874085620725da8f60a70d68de386dc"}
Sep 16 12:48:05.792052 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.791977    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75d578b874-9sjxv" event={"ID":"94a11a9d-b74b-4ae6-b35f-c6bea61d5403","Type":"ContainerDied","Data":"7a082758fff92a94f91cfe6bed75f9839281f1bdcdecf4304471b248013d05d8"}
Sep 16 12:48:05.792052 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.791999    2558 scope.go:117] "RemoveContainer" containerID="442f784ae354c3164a39c20dee3a7173b874085620725da8f60a70d68de386dc"
Sep 16 12:48:05.800825 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.800804    2558 scope.go:117] "RemoveContainer" containerID="442f784ae354c3164a39c20dee3a7173b874085620725da8f60a70d68de386dc"
Sep 16 12:48:05.801097 ip-10-0-136-98 kubenswrapper[2558]: E0916 12:48:05.801070    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442f784ae354c3164a39c20dee3a7173b874085620725da8f60a70d68de386dc\": container with ID starting with 442f784ae354c3164a39c20dee3a7173b874085620725da8f60a70d68de386dc not found: ID does not exist" containerID="442f784ae354c3164a39c20dee3a7173b874085620725da8f60a70d68de386dc"
Sep 16 12:48:05.801175 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.801095    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442f784ae354c3164a39c20dee3a7173b874085620725da8f60a70d68de386dc"} err="failed to get container status \"442f784ae354c3164a39c20dee3a7173b874085620725da8f60a70d68de386dc\": rpc error: code = NotFound desc = could not find container \"442f784ae354c3164a39c20dee3a7173b874085620725da8f60a70d68de386dc\": container with ID starting with 442f784ae354c3164a39c20dee3a7173b874085620725da8f60a70d68de386dc not found: ID does not exist"
Sep 16 12:48:05.808133 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.808108    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"]
Sep 16 12:48:05.813943 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:05.813919    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-75d578b874-9sjxv"]
Sep 16 12:48:06.325189 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:06.325153    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:07.757604 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:07.757566    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94a11a9d-b74b-4ae6-b35f-c6bea61d5403" path="/var/lib/kubelet/pods/94a11a9d-b74b-4ae6-b35f-c6bea61d5403/volumes"
Sep 16 12:48:24.735509 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:24.735474    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-774dfb4f47-jg8h6_e974f342-8f7d-46fd-a589-e9f2f7da4c44/manager/0.log"
Sep 16 12:48:24.956156 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:24.956122    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mwxr4_ca95855b-0409-4f6c-b5e2-f6ad328dfadc/manager/0.log"
Sep 16 12:48:25.184057 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:25.184029    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5_85c42150-374a-4c5c-8a55-d310175500f0/copy-manifests/0.log"
Sep 16 12:48:25.189900 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:25.189879    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5_85c42150-374a-4c5c-8a55-d310175500f0/manager/0.log"
Sep 16 12:48:25.793239 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:25.793206    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bfbbdfcb6-hgt87_f389a0e9-bea4-4a4f-ba15-138ababd43b4/maas-api/0.log"
Sep 16 12:48:26.139221 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:26.139186    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-8qnhr_69caa2c2-edeb-494f-82be-241487ed81e3/postgres/0.log"
Sep 16 12:48:27.020909 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:27.020879    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-bwqqx_53d0ab04-c567-4ee7-bfec-043856f64c9d/manager/0.log"
Sep 16 12:48:27.131367 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:27.131314    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-krfqd_7751cdd2-68e7-45e2-8f2a-32e90fc8559a/manager/0.log"
Sep 16 12:48:27.373793 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:27.372214    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-fx6rr_891012d0-fe51-436f-89e0-2b3983d51438/manager/0.log"
Sep 16 12:48:27.612774 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:27.612739    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-sg26x_3b467b85-e56c-47dc-a9c5-42b2fed0f04d/manager/0.log"
Sep 16 12:48:28.101705 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:28.101676    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wtg7l_c521403f-3dad-48da-99ad-64300ec64ecd/discovery/0.log"
Sep 16 12:48:28.325399 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:28.325370    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f67fb6dbc-hrwds_848a691d-ef56-405d-a77e-d22cb0c71c76/kube-auth-proxy/0.log"
Sep 16 12:48:28.556135 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:28.556061    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6lkxj_eed56fd2-a5fc-4aa7-a5f3-097d1a308b45/payload-pre-processing/0.log"
Sep 16 12:48:29.138029 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:29.138000    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-j6hpb_92df483b-6db3-43ce-a2e5-1f0549ff5346/main/0.log"
Sep 16 12:48:29.686074 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:29.686041    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-tzkw9_8e491c07-b1ad-4746-862f-dae40435fd76/main/0.log"
Sep 16 12:48:35.300077 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:35.300033    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:35.399320 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:35.399271    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:38.645668 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:38.645639    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xc9vl_82e3f885-f60e-4039-ac7d-3318d0fb27ca/global-pull-secret-syncer/0.log"
Sep 16 12:48:38.692366 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:38.692339    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hpmjr_b3fd6d7f-5b6f-4a27-8a8f-97e88497ffbc/konnectivity-agent/0.log"
Sep 16 12:48:38.819279 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:38.819254    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-98.ec2.internal_7698f8b509c65d280b6ab05a4a563a4f/haproxy/0.log"
Sep 16 12:48:43.272682 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:43.272647    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-bwqqx_53d0ab04-c567-4ee7-bfec-043856f64c9d/manager/0.log"
Sep 16 12:48:43.313694 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:43.313663    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-krfqd_7751cdd2-68e7-45e2-8f2a-32e90fc8559a/manager/0.log"
Sep 16 12:48:43.377916 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:43.377889    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-fx6rr_891012d0-fe51-436f-89e0-2b3983d51438/manager/0.log"
Sep 16 12:48:43.479710 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:43.479677    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-sg26x_3b467b85-e56c-47dc-a9c5-42b2fed0f04d/manager/0.log"
Sep 16 12:48:45.364155 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:45.364128    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-42wdz_6089f074-9576-4c3e-b3ae-7c59d85a1de4/monitoring-plugin/0.log"
Sep 16 12:48:45.723436 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:45.723412    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vcpsg_f106348d-5939-4254-a633-a5f44e38b5bf/node-exporter/0.log"
Sep 16 12:48:45.765286 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:45.765237    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vcpsg_f106348d-5939-4254-a633-a5f44e38b5bf/kube-rbac-proxy/0.log"
Sep 16 12:48:45.795637 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:45.795606    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vcpsg_f106348d-5939-4254-a633-a5f44e38b5bf/init-textfile/0.log"
Sep 16 12:48:45.825050 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:45.825018    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4cvlw_11685ce0-7e31-48ff-b729-dfe81f8cfcde/kube-rbac-proxy-main/0.log"
Sep 16 12:48:45.846461 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:45.846433    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4cvlw_11685ce0-7e31-48ff-b729-dfe81f8cfcde/kube-rbac-proxy-self/0.log"
Sep 16 12:48:45.878392 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:45.878322    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4cvlw_11685ce0-7e31-48ff-b729-dfe81f8cfcde/openshift-state-metrics/0.log"
Sep 16 12:48:46.243649 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:46.243568    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-658d964cd-d8l8g_8b64bb41-3b66-43ff-9288-52dd90d7e6a1/telemeter-client/0.log"
Sep 16 12:48:46.267338 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:46.267306    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-658d964cd-d8l8g_8b64bb41-3b66-43ff-9288-52dd90d7e6a1/reload/0.log"
Sep 16 12:48:46.295407 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:46.295377    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-658d964cd-d8l8g_8b64bb41-3b66-43ff-9288-52dd90d7e6a1/kube-rbac-proxy/0.log"
Sep 16 12:48:48.760444 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.760408    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"]
Sep 16 12:48:48.760867 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.760798    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a11a9d-b74b-4ae6-b35f-c6bea61d5403" containerName="maas-api"
Sep 16 12:48:48.760867 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.760812    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a11a9d-b74b-4ae6-b35f-c6bea61d5403" containerName="maas-api"
Sep 16 12:48:48.760948 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.760881    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="94a11a9d-b74b-4ae6-b35f-c6bea61d5403" containerName="maas-api"
Sep 16 12:48:48.764463 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.764439    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.767176 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.767149    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2psvv\"/\"kube-root-ca.crt\""
Sep 16 12:48:48.767334 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.767196    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2psvv\"/\"default-dockercfg-nq47s\""
Sep 16 12:48:48.767886 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.767867    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2psvv\"/\"openshift-service-ca.crt\""
Sep 16 12:48:48.775692 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.775669    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"]
Sep 16 12:48:48.818015 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.817977    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-podres\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.818015 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.818025    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-lib-modules\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.818268 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.818087    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp2sr\" (UniqueName: \"kubernetes.io/projected/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-kube-api-access-gp2sr\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.818268 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.818116    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-sys\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.818268 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.818188    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-proc\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.919312 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.919273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp2sr\" (UniqueName: \"kubernetes.io/projected/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-kube-api-access-gp2sr\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.919512 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.919324    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-sys\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.919512 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.919455    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-proc\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.919663 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.919522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-podres\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.919663 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.919562    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-lib-modules\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.919663 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.919581    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-sys\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.919663 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.919610    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-proc\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.919826 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.919701    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-podres\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.919826 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.919720    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-lib-modules\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:48.928371 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:48.928344    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp2sr\" (UniqueName: \"kubernetes.io/projected/14cb6dad-fdad-48a7-bbcc-d90c2569dfa0-kube-api-access-gp2sr\") pod \"perf-node-gather-daemonset-h54c8\" (UID: \"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:49.074954 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:49.074858    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:49.097417 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:49.097379    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c5bf78c54-692zz_ab6b0364-809b-407a-b8e2-d949c6316add/console/0.log"
Sep 16 12:48:49.348783 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:49.348756    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"]
Sep 16 12:48:49.351295 ip-10-0-136-98 kubenswrapper[2558]: W0916 12:48:49.351268    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod14cb6dad_fdad_48a7_bbcc_d90c2569dfa0.slice/crio-17dbbdb8112fa71dcb3b43957d20fb5ccfa493f85c2cb8085b568fdbc9e8ee6d WatchSource:0}: Error finding container 17dbbdb8112fa71dcb3b43957d20fb5ccfa493f85c2cb8085b568fdbc9e8ee6d: Status 404 returned error can't find the container with id 17dbbdb8112fa71dcb3b43957d20fb5ccfa493f85c2cb8085b568fdbc9e8ee6d
Sep 16 12:48:50.004409 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:50.004375    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8" event={"ID":"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0","Type":"ContainerStarted","Data":"af80997d2e7b93e52073ca6d7c9fdedefab01f87d7a3814b08ceb68723517494"}
Sep 16 12:48:50.004836 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:50.004428    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8" event={"ID":"14cb6dad-fdad-48a7-bbcc-d90c2569dfa0","Type":"ContainerStarted","Data":"17dbbdb8112fa71dcb3b43957d20fb5ccfa493f85c2cb8085b568fdbc9e8ee6d"}
Sep 16 12:48:50.004836 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:50.004506    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:50.029435 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:50.029381    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8" podStartSLOduration=2.029365179 podStartE2EDuration="2.029365179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:50.029111396 +0000 UTC m=+2726.783070128" watchObservedRunningTime="2026-09-16 12:48:50.029365179 +0000 UTC m=+2726.783323951"
Sep 16 12:48:51.116643 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:51.116612    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xz5cp_1f341464-bbd6-4a15-82fe-51149ded0036/dns/0.log"
Sep 16 12:48:51.143077 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:51.143055    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xz5cp_1f341464-bbd6-4a15-82fe-51149ded0036/kube-rbac-proxy/0.log"
Sep 16 12:48:51.195085 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:51.195059    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fk9r5_678bc08c-e65a-4d2f-8de1-5b73269448c2/dns-node-resolver/0.log"
Sep 16 12:48:51.740626 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:51.740597    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-68fc5bf8bf-4d95z_22f72982-d747-49aa-81e6-f274ce50bb39/registry/0.log"
Sep 16 12:48:51.809295 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:51.809265    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tgspg_4c0a1d8d-4b3a-499b-ab3e-4426fcb5948b/node-ca/0.log"
Sep 16 12:48:53.299448 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:53.299407    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wtg7l_c521403f-3dad-48da-99ad-64300ec64ecd/discovery/0.log"
Sep 16 12:48:53.347990 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:53.347956    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f67fb6dbc-hrwds_848a691d-ef56-405d-a77e-d22cb0c71c76/kube-auth-proxy/0.log"
Sep 16 12:48:53.414993 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:53.414959    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6lkxj_eed56fd2-a5fc-4aa7-a5f3-097d1a308b45/payload-pre-processing/0.log"
Sep 16 12:48:54.405347 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:54.405309    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sshpk_3ef9d7a4-3c9d-4802-8871-18079adb56ea/serve-healthcheck-canary/0.log"
Sep 16 12:48:55.173367 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:55.173343    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ts76n_1e51c0fd-41e1-40df-9051-d41529de1881/kube-rbac-proxy/0.log"
Sep 16 12:48:55.195977 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:55.195950    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ts76n_1e51c0fd-41e1-40df-9051-d41529de1881/exporter/0.log"
Sep 16 12:48:55.218041 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:55.217999    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ts76n_1e51c0fd-41e1-40df-9051-d41529de1881/extractor/0.log"
Sep 16 12:48:56.020528 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:56.020497    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-h54c8"
Sep 16 12:48:57.783695 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:57.783668    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-774dfb4f47-jg8h6_e974f342-8f7d-46fd-a589-e9f2f7da4c44/manager/0.log"
Sep 16 12:48:57.885816 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:57.885782    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mwxr4_ca95855b-0409-4f6c-b5e2-f6ad328dfadc/manager/0.log"
Sep 16 12:48:57.989305 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:57.989272    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5_85c42150-374a-4c5c-8a55-d310175500f0/manager/0.log"
Sep 16 12:48:58.017902 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:58.017877    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5b57bf8b-7wvk5_85c42150-374a-4c5c-8a55-d310175500f0/copy-manifests/0.log"
Sep 16 12:48:59.509996 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:59.509956    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wtg7l_c521403f-3dad-48da-99ad-64300ec64ecd/discovery/0.log"
Sep 16 12:48:59.541846 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:59.541818    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f67fb6dbc-hrwds_848a691d-ef56-405d-a77e-d22cb0c71c76/kube-auth-proxy/0.log"
Sep 16 12:48:59.587556 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:48:59.587510    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6lkxj_eed56fd2-a5fc-4aa7-a5f3-097d1a308b45/payload-pre-processing/0.log"
Sep 16 12:49:01.543850 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:49:01.543812    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wtg7l_c521403f-3dad-48da-99ad-64300ec64ecd/discovery/0.log"
Sep 16 12:49:01.584566 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:49:01.584522    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f67fb6dbc-hrwds_848a691d-ef56-405d-a77e-d22cb0c71c76/kube-auth-proxy/0.log"
Sep 16 12:49:01.629217 ip-10-0-136-98 kubenswrapper[2558]: I0916 12:49:01.629183    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6lkxj_eed56fd2-a5fc-4aa7-a5f3-097d1a308b45/payload-pre-processing/0.log"